Posts Tagged ‘QA’

Test Escape Reports

We just had a big launch at work, which of course leaves me thinking about test escape reports.

I have mixed feelings on test escape reports, and for a long time I didn’t want to use them here. My reason being that if you have a small test team, and you know everyone is really dedicated and working hard, then the occasional test escape is to be excepted and we shouldn’t harp on them.

However, I think over time I’ve seen the other side of this issue. Test Escape reports ensure stakeholders that you are on top of any issues both before launch and after. They hold the team accountable, not just for the issue but also for the resolution and prevention of it in the future.


The format we use is simple:



Reason missed:

Steps taken to cover functionality:

Test cases numbers for verification:


The point being not to look backward, or to blame someone for missing a bug- but to look forward into how we can test that in the future and how we can avoid over looking similar areas on other projects. The goal is to improve visibility- we admit our mistakes, take steps to correct them and let everyone take comfort in/participate in the improved coverage.

How do you handle missed bugs? Are test escapes’ a whole team problem or an individual tester’s problem? Is there a formal process surrounding them?


Become a Better QA

What do you do to become a better QA?

It is easy to get distracted by the rush of every day work as a tester. Over time, you get practice and your skills develop and you progress. But it seems such a passive way to go about improving yourself and your career. There are numerous books and blogs and communities out there, not to mention whatever is available locally in your area. So how do you chose?

When I started at my current job, I made it an ongoing personal goal to learn as much as I could about testing and related technologies that will make my job easier, make my team stronger and help our testing efforts go further.

In the last year, I have met a lot of mini-goals along that path- becoming  more comfortable with shell scripting, starting to learn ruby, reading at least two testing books per quarter – but I still feel like I have a lot to do! It is both exciting and overwhelming to dive into open-ended self education.

I frequent blogs that put out fresh new ideas and share experiences, I follow groups online and attended an awesome conference. I’ve considered taking evening classes at a community college (I loved my evening ruby class last spring) and trying to get more involved with the local community. I think both of those may come into play next year. My focus for the next year is both on strengthening my agile testing skills as we have teams switching over to SCRUM, as well as to continue learning Ruby and of course, reading about new QA techniques and approaches.

But where do you turn to become a better tester? How do you decide what to work on and where to improve your skills?

What I’ve learned about Agile Testing

I started on an agile team last August. In the past few months, I’ve be exposed to a wide range of agile-ness as we have learned to adapt it for different projects and struggled to make it work for us. I have found great worth in some of its practices, and unfinished mess in others. As a team, the company has had to switch practices and courses a few times. The cool thing is that we are able to. No one is so stuck on procedure as to not admit that it could use tweaking. And where procedure was too lax, people have asked for more.

Here is what I’ve learned as the QA department has grown up and around the development teams:

– Agile doesn’t mean “No Processes” it just means flexible processes

Documentation is more a struggle. Shortcuts made in the name of agileness and flexiblity have been huge road blocks in my testing effort. We are still sorting out the right place for them. Something that allows for creativity and adjustments, but still leaves us with a solid document to test against.

– Trying to keep meetings short by cutting out discussion is pointless and unhelpful. Structuring meetings to allow for quick agendas and then discussion time is priceless.

Transparency between departments and code reviews are QA’s best friend, even if it takes more time at first.

– Short iterations mean quick test cycles. This means you need to have as much ready before had as possible – test plans, test suites – both automated an manual, report templates etc can all help keep you on track.

– Managing an agile team means knowing when to step back. The team is responsible to each other, not to you.

– Be flexible. This should be obvious, and maybe it is to some. But in my experience, learning agile meant learning to contort your testing efforts to the time and space available. That doesnt mean you should let your team get pushed around, but it means that you have to be willing to do the agile dance along with everyone else.

– QA needs to make its needs known. Don’t let yourself get deadlines pushed back and back until you are the one left struggling to pick up all the slack. Just because deadlines can change mid cycle now, doesn’t mean that your team has to sacrifice thoroughness in the name of changing requirements and speed.

– Communication is even more important – with things changing and people having so much freedom with their work, if you have a bad communicator, your whole project is at risk.

I am sure this list will grow in the coming months. For those of you who are new to Agile testing- what have you learned? Was it what you expected? Have there been any roadblocks on your project?

Top QA blogs listed

Testingminded listed the top 100 QA blogs here.  There are a ton of good blogs out there and it is great to see some centralized lists.

Here are the top 5:

Make sure to  check out his page to see the rest!

Write bugs that get fixed

It seems simple enough- find an issue, file a bug, wait for it to be fixed. The tough part should be finding the bug in the first place, but sometimes it isn’t.

1- Time constraints. When deadlines loom, the team may be anxious to get the project out the door and ignore “minor” issues that can be fixed post launch.

2-Complexity. Even simple sounding bugs have have complex integration with other functionality or require a lot of rework – and no one likes rework.

3- Dismissal. No one likes to hear this one, but it happens. Sometimes bugs get dismissed because a) the developer doesn’t trust the QA or hasn’t seen the issue b) it feels like a rare corner case c) it is hard to reproduce d) there are a lot of other demands on the team at the time and your bug simply falls off the radar.

How can we avoid these issues?

We can write clear bugs. Reproduction steps are one of the strongest tools we have. One of my favorite QA interview questions involves getting candidates to write reproduction steps for an unexpected action. It is a really important skill for QAs. Project managers and engineers need to really understand how we got to the issue we did and be able to see it themselves- not to mention how much it helps us if we have to explain our bugs to someone.

We can follow up on issues. How many times have you written a bug just to let it slip by unnoticed while you found more issues? Checking on the status of issues isn’t as exciting as finding new ones, but it is important if you are trying to get something fixed. If a bug gets put in deferred, make sure you know why. If it is fixed- does the fix match the documents?

We can be knowledgeable. Sure, we aren’t developers- but it helps if we know about about the software they are writing. If there are lots of false alarm bugs or low priority issues labeled as high priority, the team will start to trust us a little less. Knowing about the documents, the software and the issue make you a much more creditable source and seems to help your bugs get fixed.

We can “file” some bugs off the record. I’m not really promoting this- especially if your company is big on formal policies. But, if it makes sense for your team, letting developers know about smaller issues first, offline, not only gives them a head start, it puts us all on the same team. We aren’t filing bugs against developers, we are working with them to make sure everything runs right.

Does anyone else have tips for making sure bugs get addressed?

Article in TEST!

I have an article in this month’s TEST magazine called: Keep the User in Mind. You can find it on their website here, but I will paste the text below, also. Also in the March issue, they are running the completed “High Tech in High Heels” series from this blog, so keep an eye out! I am very excited to contribute to this awesome magazine.

Keep the user in mind

Devon Smith* QA Lead at Agora Games LLC in New York says you need to keep the user in mind, especially as you approach release time.

It’s late on the last day of testing and your eyes are a little blurry from some last minute verifications. Your bug list is longer than you expected, and the coffee in the break room is long since burnt. Everyone is ready for you to sign off and you can hardly remember why that final test really matters. After the meetings have been held, requirements agreed upon, and countless lines of code written and implemented, QA testing can seem like a roadblock to the exciting part – the great release. But it is exactly in that moment, with a stale cup of coffee in your hand and one last run through to do on the software, that you most need to remember who you answer to: the user.

Each team has different priorities in a release, and sometimes, that leaves Quality Assurance pulled in a lot of different directions. On one side there are the project manager’s deadlines, on another are ambiguities in the requirements document, and on still another is the cool new engineering solution the developers are trying out. Working between the different departments is part of what makes QA challenging and interesting. Understanding and balancing the motivations and needs of each team helps us to do a better job, but the people we really need to understand are the users. We need to understand them to recognize the limitations of the software, to know how business goals will be met, and to appreciate how real people will use the fancy designs and complex solutions. That is our angle. Our goal is to focus on the user. Keeping the user in mind while we test helps keep us rooted in the real world, helps us to stay focused, and helps us deliver that quality release.

We may verify everyone elses’ work, but the users verify ours. They keep us accountable. Often, we are encouraged to test in a hurry, keep our bug counts high, and track the severity of our defects. Those are just outward measures of what is really important. On that last night of testing, when debugging and manual tests seem endless, remember that you are the last stop between the software and the user. It is not about the bug counts, or documents or deadlines – it is about watching out for the users, who truly are the final sign off on quality.

*Devon Smith is the QA Lead at Agora Games LLC in New York. She has been working in software testing for three years, has a bachelors degree from the University of Washington in Seattle and worked at Sun Microsystems before coming to Agora. Her feature, High tech in high heels appears in the March issue of T.E.S.T magazine.


Now that choosing a new team member is complete, it is time to formalize goals for the QA department moving forward. When I started here, QA was a footnote to the development process. The engineers ran unit tests and the production team “clicked through” some functionality at the end. But no one was really focused on what a QA department should look like or what that role should be. I think one of the biggest challenges I faced when I came on was carving out a bigger spot for QA within the existing environment. It is an ongoing process, but in the past six months, I’ve seen vast improvement. It is amazing to see processes start to take shape and make an impact. This position has been incredibly rewarding, in a large part, because I can see improvement every day. The whole team here has been flexible, even when I tried something that didn’t really fit, everyone gave me time to try it and then quickly adapted to something new until we found the processes that worked for us. It is nice to reflect on that progress. But no matter how much further we are now than when I started, there is still so much that can be improved upon.

What I hope to accomplish next is opening up the QA process more to the development team. There is a review process now,  but it is loosely followed. I’d like to finish a test plan and have a round table time for people to review, critique and discuss risk areas and test cases. I think it could help in so many facets. Engineers can think more about risk areas when programming and QA can hear about trouble spots or areas of confusion and revise test cases to be more complete in those areas. The production team will have a more complete picture of what needs to go on before a deploy and most importantly, we all get a chance to sit and talk about quality together. I think its really important for such a discussion to occur before the official testing period.

I keep coming back to where QA teams can add value to the whole process, not just the end product. In an interview, I had one candidate define QA’s role as “Reactionary” to the development process. I have to say, I really don’t want that in this department. I want the QA team to be integrated and active throughout the development process – proactive about reviews, discussions, risk areas and testing. There is no reason to wait until the end of a cycle for that, especially not on an agile team. So here is to a larger team, new goals, and hopefully some more great progress like we have seen in the past six months.