Archive for the ‘Teamwork’ Category

Test Escape Reports

We just had a big launch at work, which of course leaves me thinking about test escape reports.

I have mixed feelings on test escape reports, and for a long time I didn’t want to use them here. My reason being that if you have a small test team, and you know everyone is really dedicated and working hard, then the occasional test escape is to be excepted and we shouldn’t harp on them.

However, I think over time I’ve seen the other side of this issue. Test Escape reports ensure stakeholders that you are on top of any issues both before launch and after. They hold the team accountable, not just for the issue but also for the resolution and prevention of it in the future.


The format we use is simple:



Reason missed:

Steps taken to cover functionality:

Test cases numbers for verification:


The point being not to look backward, or to blame someone for missing a bug- but to look forward into how we can test that in the future and how we can avoid over looking similar areas on other projects. The goal is to improve visibility- we admit our mistakes, take steps to correct them and let everyone take comfort in/participate in the improved coverage.

How do you handle missed bugs? Are test escapes’ a whole team problem or an individual tester’s problem? Is there a formal process surrounding them?


Having interns on the QA team

Internships are good for the interns, right? As an intern, you get a chance to learn, to be in a professional environment, sometimes- class credit and of course, some of your first professional references.  Interns get to be around the processes they are learning about first hand, they get to attend daily standups, read through requirements and work on project deadlines. It can be a great experience, but not just for the interns. The whole department can benefit from having some amazing interns around.

Having interns on the team has brought a different point of view. While they don’t have as much professional work experience they do know about the process and are used to learning new things quickly. I’ve been impressed with the quality of interns we have gotten. It is also great to have people with a little more programming knowledge who can help us make adjustments and revisions to tools.

Managing interns provides many of the same complications that managing a regular team of full time people does, but with some added quirks. The biggest quirk is that you have to realize that this job is not their focus. Sure, you spend 40 hours a week at it and it is your career. It is really important to you. But for them, at best this is part of their education, and at worst, it is an after school job. School comes first. Finals week, family vacations, homework- all of that is higher priority. Which is as it should be.

Also, they aren’t used to the office. It can be overwhelming. I personally think this is one of the best advantages an internships can provide, because they can become more comfortable with a professional setting early. Usually this isn’t a problem, but when assigning out projects or attending meetings, it is a good thing to remember.

Lets be honest, for some reason, not many people go to college to be a software QA. There aren’t really any programs and a lot of computer science programs don’t even include it as a required class. Want to get people excited about QA? Why not include it as part of the process while you are studying in college.

I like to think that I’m helping out future QAs by teaching future devs that the QA department does work hard. We experiment with new tools and new languages until we find the right mix for the project, we know how to get around the command line and can read and write some code, we pour over requirements documents and offer suggestions and ask questions. We act as user advocates, we clarify stories, keep long suites of tests debugged and running. We know there is limited time and so we test quickly, doing our best to squeeze out as much testing as we can in what little extra time there is in a project. I jokingly tell them (but not so jokingly) to remember all that when they are leading a Dev team some day.

Time for QA to fully embrace Agile

My favorite idea from Agile 2010 was that “QA should be the headlights for a project” A quality guide, not a quality police.

We can sit at the end of the schedule, run tests and file bugs. Or, we can jump out in front, help define requirements, learn from the programmers as they work on the project, offer advice and keep the user experience and project quality on everyone’s mind from beginning to end. Which helps the project more? Actually, which is more fun? Because personally I’d much rather get involved and get people excited about turning out a good product in the end, than just force quality on them right before release.

So how do you change your company culture to embrace that? I am presenting at a lunch and learn next week, with the hopes of kicking off this next phase of the QA process. I am so inspired and excited for the change after a week of being surrounded with agile talks… but how will the company take it? I think I have gotten my department on board, but the hard part will be the other departments who have to put up with us a lot more now. No one likes change, and organizations are harder to change than people.

So how do you get everyone excited about it? How do you show everyone how important this will be? I haven’t finished my presentation yet, but I’m trying to think back to those really inspiring sessions at the Agile Conference and give it a go. Heres hoping!

Extending Lighthouse bug tracking

We use lighthouse for bug tracking at my company. It is simple, flexible and has a great api that allows us to hook it into testlink to track bugs from our test management system. The search functionality, however, seems limited. I have often wished there were a dashboard feature that allowed for more of an overview. It might be different if I worked on only one project, but as a QA lead,  I care about all the projects all the time. So when my company decided to do a hackathon day, I wanted to find a way to extend lighthouse into a dashboard.

We had 8 hours and 4 people to complete our hackathon task. Since a number of us had just completed a basic intro to Rails class, we decided to make a separate page that pulls data from Lighthouse. We were able to get the info out of lighthouse pretty easily with the API and Rails (with web themes) let us get a basic page up quickly. With so much data coming in, we had to figure out how to paginate the results so that the system didn’t run into any lag while trying to display the results. And the profile creation and filtering process took us a while.

By the end of the day, we had a working profile up with dynamic info coming in from our Lighthouse projects. A user can log in and see a list of their current tickets for all projects, with priority and status displayed and link those back to lighthouse to update them, or they can go to a more project specific view and see tasks assigned to them – with details for each project.

We have more work to do- we want to work on the sorting so that tickets appear in different bins by status, by priority, date updated etc. We also want to make a “team view” so that leads can log in and see the release status for all the members of their team. Eventually, we hope to do similar integration with testlink to display test status, test cases assigned and test cases created by you. We also want to integrate with Pivotal Tracker, which is also great from the project perspective, but needs a cross- project dashboard for those of us who are always working cross- team.

When Low Priority Bugs Should be Fixed

So you file a UI issue. No functionality is broken, the site looks ok. The deadline is close, and it makes sense to launch it. No one wants to delay a deploy because of a few pixels. When do you put your foot down? Maybe it isn’t worth fixing the color on a banner that only shows up for 5% of people using the application, or optimizing for an outdated browser that few people use. But not all issues that are marked as low priority should be ignored.

Low priority issues should be fixed when they make you look sloppy. Sure, everything works, but if your alignment is off, your headlines are the wrong font size, or spelling / grammar could be improved – the overall look of the application is sloppy. People will not remember that the log in process worked just fine, they will remember that your news story hung halfway off the page, that they had to scroll down the page to get to information or that your content blocks were not lined up.

low priority defect on Facebook

Low priority issues should also be fixed if they they add frustration (now or in the future). If your page can only be viewed well in one or two browsers, you may lose users who don’t want to download and use an unfamiliar browser.  Anything that can cause frustration to your users or waste the customer service department’s time helping people find work around is better off fixed.

Your site should be polished- functionality, and look and feel. If adjusting a low priority defect gives a more professional overall appearance, or saves you from having to have a detailed troubleshooting page- they should be fixed before launch. And it is worth the QA departments’ time to make a stand for them.

Developers: A Ray of Sunshine

Developers may be the most blatantly, unashamedly  optimistic group of people I have ever known. In the face of all evidence to the contrary, they believe that this time, software will work, deadlines will be met, solutions will be easy and testing will find no bugs.  It doesn’t matter how quickly something was put together or how shaky the requirements, or how new the system – developers feel completely confident that things will be fine. This never ceases to amaze me. I hate being the rain cloud on their bright day, but at the same time, I do enjoy injecting a little realism.

Code freeze just two days before launch? No problem.

Late requirements? We can squeeze them in!

last minute fixes? It shouldn’t be too hard/ (and my favorite) This will just take a minute.

They deploy their hard work off to staging, crack open a beer, and sit in the sun while testing begins. Even before you start that first test, you know it is too early for that beer. You remember last release, and the one before that. You remember that “quick bug fix” that took two days of testing and reworking to straighten out. You know this time is not different. You know they have deployed code that is broken, and that in a matter of time and tests, you will find the first broken spot- and then the next. You know that no matter how long or how hard you work, when testing is over and the project goes live, there will still be more bugs uncovered days, weeks or months later. You can only hope that if you work really hard, you will catch all the big issues. But that is your job- to remember all the ways things have broken before.

Developers spend their days creating new systems and solving problems. They get instant gratification when they do little fixes and get their unit tests to pass. They thought up their own solution, and so are somewhat blinded to potential issues  by their very involvement. Software testers spend their days slogging through broken programs, early releases, and failed bug fixes. Once something is fixed, we stop using it. We read countless bug reports and user issues and spend our days writing test cases trying to find broken spots. We aren’t involved in that creative burst of fixing something or designing something. Our creativity comes from all the ways we find to break it.

So when you file that first bug of the testing cycle, remember… their optimism will come in handy when it is time to fix all the issues you find. Besides, we are optimistic in our own way. We fully believe that their are bugs out there to find, and each day, we are proved blissfully right. Each issue we file a little victory, because that is one less issue for there to be in production, which helps the developers, QA and project managers to feel more optimistic about the release as a whole.

Write bugs that get fixed

It seems simple enough- find an issue, file a bug, wait for it to be fixed. The tough part should be finding the bug in the first place, but sometimes it isn’t.

1- Time constraints. When deadlines loom, the team may be anxious to get the project out the door and ignore “minor” issues that can be fixed post launch.

2-Complexity. Even simple sounding bugs have have complex integration with other functionality or require a lot of rework – and no one likes rework.

3- Dismissal. No one likes to hear this one, but it happens. Sometimes bugs get dismissed because a) the developer doesn’t trust the QA or hasn’t seen the issue b) it feels like a rare corner case c) it is hard to reproduce d) there are a lot of other demands on the team at the time and your bug simply falls off the radar.

How can we avoid these issues?

We can write clear bugs. Reproduction steps are one of the strongest tools we have. One of my favorite QA interview questions involves getting candidates to write reproduction steps for an unexpected action. It is a really important skill for QAs. Project managers and engineers need to really understand how we got to the issue we did and be able to see it themselves- not to mention how much it helps us if we have to explain our bugs to someone.

We can follow up on issues. How many times have you written a bug just to let it slip by unnoticed while you found more issues? Checking on the status of issues isn’t as exciting as finding new ones, but it is important if you are trying to get something fixed. If a bug gets put in deferred, make sure you know why. If it is fixed- does the fix match the documents?

We can be knowledgeable. Sure, we aren’t developers- but it helps if we know about about the software they are writing. If there are lots of false alarm bugs or low priority issues labeled as high priority, the team will start to trust us a little less. Knowing about the documents, the software and the issue make you a much more creditable source and seems to help your bugs get fixed.

We can “file” some bugs off the record. I’m not really promoting this- especially if your company is big on formal policies. But, if it makes sense for your team, letting developers know about smaller issues first, offline, not only gives them a head start, it puts us all on the same team. We aren’t filing bugs against developers, we are working with them to make sure everything runs right.

Does anyone else have tips for making sure bugs get addressed?