Litmus: Friend or Foe

I use Litmus quite extensively for cross-browser testing. We have done some email testing with it, but not nearly as frequently as we use it for basic site testing.
In general the concept is great. Simple, easy to use, clean. When it works, it provides useful screen-shots from almost any browser of your choice

Its email testing client is also awesome. This service feels more robust and has been improving in the past year and a half or so I’ve been using Litmus. ( in fact, I wrote an earlier review of Litmus when I first started using it- but as time went by, I felt I had more to add to a review of the tool than I did initally) It seems to me the company may be moving in this direction. While I do like the email tests, I hope they don’t abandon page testing all together.

starting a new page test for cross browser testing


But on the Website testing front, can litmus be used reliably for cross browser results? My biggest complaint is that anything on staging loads so slowly, that often whole sections of the page don’t display properly. It also only clears one level of basic auth, so sites that require more than that will not run.
I think with a few tweaks and clean ups, it could offer a much more robust service to what is already a useful too.
For example, I can send a link to the screenshot result, but I can’t annotate it. Wouldn’t it be great to be able to point out specific areas of concern to developers in a neat, clean way?

Not all browsers offer full page scrolling. That means that sometimes, only the top of the page is visible on the test result.

Offer better support for mac browsers. For some reason, mac browsers always come back with an error and you have to run them multiple times to get them to work. This limits the effectiveness of the test suite.

Overall, this is a useful tool, and I haven’t found anything better to get in full cross-browser testing. However, it does have a few short comings that make working with it more cumbersome and less useful than it could be. It does seem to be getting updates and developing- so hopefully some of these issues will get addressed.

Does anyone else use this tool? If not, what have you found for cross browser testing and do you find it effective?


New Year, New Site

So for 2011, this blog will be getting a makeover. New title, new theme, new posts and last but not least- a new address. I have decided to try hosting my own blog. Does one successful Rails class and an automated test suite qualify me to host my own blog? I guess we’ll find out!
I spent the last weekend settling on a domain name, Everyday QA, migrating my old posts and comments over and learning some basic html editing. I will be setting up forwarding from this site once I’m ready to do the big switch. For now, head on over to the new site- and let me know what you think! I’d love feedback on how it looks and what should be changed.

I hope you all enjoy the new site and I look forward to getting lots of new posts and information up!

Job Opening in the QA department

We’re hiring for a new QA team member at Agora! I’ll post the job info here, in case anyone knows of a great candidate. Check out our website for more information about life at Agora.

Quality Assurance Analyst

Agora Games (, a subsidiary of Major League Gaming (, develops online video game technology and web communities for game developers and publishers. Our technology makes compelling information available to players both in-game and online, while providing publishers with a wealth of demographic information. Over 28 million unique players are tracked by our system and billions of bits of data are spread across a cutting edge virtual computing platform. Our portfolio includes work on titles including Guitar Hero, Call of Duty, and Transformers, and we cross all the major gaming platforms.

Agora is seeking a Software Quality Assurance Analyst to join our QA team. We test software for MLG’s websites and our proprietary programs, designed to enhance the community experience around some of the top video games. In this position you will be expected to work closely with developers in an Agile environment to deliver quality products. You will provide feedback about response under projected load, risk areas, and overall experience of the application.


Duties and responsibilities:

Automated and manual and load testing using Selenium, Siege, Jmeter

Write and execute test cases

Keep automated tests up to date and debugged

Provide feedback on features during sprints

Ambiguity Review of requirements and technical documentation

Report detailed and accurate bugs and follow them through to verification

Interpret requirements from various sources and apply basic testing philosophies and heuristics to test software at different stages of the development process.


* At least 1 year experience in the software QA field

* Experience testing with Selenium, or other automated testing tools.

* Basic experience with a scripting language

* Familiarity with unix

* Interest in working on-site at our office in Troy, NY.

The ideal candidate will have:
* Attention to detail

* Analytical mindset

* Experience testing web properties

* Love video games!

* A proactive mindset and be able to multitask and prioritize requirements.

* Experience with Agile development methodologies

* Must be self-motivated, posses excellent communication skills, and be a team-player.

* Competitive salary and benefits

* Work with an experienced, successful, intelligent team

* Be an integral part of a growing, fast-paced company where your work will make a true impact

All applicants should submit their resume and cover letter.

We want to hear from you! Agora values excellence, passion, integrity and creativity. Submit your application to

Test Escape Reports

We just had a big launch at work, which of course leaves me thinking about test escape reports.

I have mixed feelings on test escape reports, and for a long time I didn’t want to use them here. My reason being that if you have a small test team, and you know everyone is really dedicated and working hard, then the occasional test escape is to be excepted and we shouldn’t harp on them.

However, I think over time I’ve seen the other side of this issue. Test Escape reports ensure stakeholders that you are on top of any issues both before launch and after. They hold the team accountable, not just for the issue but also for the resolution and prevention of it in the future.


The format we use is simple:



Reason missed:

Steps taken to cover functionality:

Test cases numbers for verification:


The point being not to look backward, or to blame someone for missing a bug- but to look forward into how we can test that in the future and how we can avoid over looking similar areas on other projects. The goal is to improve visibility- we admit our mistakes, take steps to correct them and let everyone take comfort in/participate in the improved coverage.

How do you handle missed bugs? Are test escapes’ a whole team problem or an individual tester’s problem? Is there a formal process surrounding them?

Having interns on the QA team

Internships are good for the interns, right? As an intern, you get a chance to learn, to be in a professional environment, sometimes- class credit and of course, some of your first professional references.  Interns get to be around the processes they are learning about first hand, they get to attend daily standups, read through requirements and work on project deadlines. It can be a great experience, but not just for the interns. The whole department can benefit from having some amazing interns around.

Having interns on the team has brought a different point of view. While they don’t have as much professional work experience they do know about the process and are used to learning new things quickly. I’ve been impressed with the quality of interns we have gotten. It is also great to have people with a little more programming knowledge who can help us make adjustments and revisions to tools.

Managing interns provides many of the same complications that managing a regular team of full time people does, but with some added quirks. The biggest quirk is that you have to realize that this job is not their focus. Sure, you spend 40 hours a week at it and it is your career. It is really important to you. But for them, at best this is part of their education, and at worst, it is an after school job. School comes first. Finals week, family vacations, homework- all of that is higher priority. Which is as it should be.

Also, they aren’t used to the office. It can be overwhelming. I personally think this is one of the best advantages an internships can provide, because they can become more comfortable with a professional setting early. Usually this isn’t a problem, but when assigning out projects or attending meetings, it is a good thing to remember.

Lets be honest, for some reason, not many people go to college to be a software QA. There aren’t really any programs and a lot of computer science programs don’t even include it as a required class. Want to get people excited about QA? Why not include it as part of the process while you are studying in college.

I like to think that I’m helping out future QAs by teaching future devs that the QA department does work hard. We experiment with new tools and new languages until we find the right mix for the project, we know how to get around the command line and can read and write some code, we pour over requirements documents and offer suggestions and ask questions. We act as user advocates, we clarify stories, keep long suites of tests debugged and running. We know there is limited time and so we test quickly, doing our best to squeeze out as much testing as we can in what little extra time there is in a project. I jokingly tell them (but not so jokingly) to remember all that when they are leading a Dev team some day.

Moving to Agile: What the QA team needed to learn.

When my company moved to  Agile, the QA team had to make some adjustments to integrate and fit in with the new process. We’ve been working at it for a good three months now, and I think we are all starting to feel more comfortable with it.

Staying in Step:

Firstly, QA is used to being just a little behind development. We were focused on projects, not features. Although we tried to get in as early as possible, we were usually still an iteration or a milestone behind. It wasn’t terrible, but there were complaints that by the time we had questions about features or stories, the dev team had already moved on and answering our questions meant that they had to switch gears. We had a hard time because it seemed like our bugs stuck around longer and we were just a little out of the loop. Playing catch up on projects when we came in and ramped up was hard and time consuming. We wanted more communication.

Now, we attend each development team’s daily stand up – so we not only are held accountable for our tasks but we hear what the team is working on from day to day. We are part of the sprint planning meeting and we work along side the developers so our questions usually come up early enough to save time and make clarifications right when the feature is being built.

Divide and Conquer:

We have a few teams each working on different projects. Its a lot for one QA to keep up with.  I guess one of the challenges of a small QA team is finding a balance between giving each project the focus and dedication needed, while still getting all the work done on other projects. To handle this,  I split up to cover more. We can each specialize in one specific team, work with them, attend their meetings, and stay up to speed on all their projects. Our own, informal standups just allow the QA team to give eachotehr a quick overview.  It works because each QA gets to be an expert on a few projects and has time to get more indepth when they need, but if they need to switch projects to help out, there is already an expert on that project ramped up and ready to help.


Accept Early:

My favorite thing about the switch to Agile is acceptance tests. Once each story is defined, QA works with developers to come up with acceptance steps that become part of our story. This has been great, because I get to talk to the team before they start working to clear up any ambiguities about the feature. I also get a heads up so that when they finish it, I already know what to expect and how I want to test. It has increased our team work and our communication and it helps QA stay out in front of the game.  Once all the features have been looked at individually, we give the whole product a run through, of course. But it gives everyone piece of mind to have the features tested individually first.

Ruby: A non-programmer’s foray into a programming language.

In high school, English was my thing. (What does this have to do with Ruby? I’m getting there.) I loved picking up a pen and chasing down words to discuss novels or detail moments or to wax poetic about the political structure of  Rome. Have you ever watched someone make pottery on a wheel? Their fingers sliding over wet clay, pressing in slightly and urging a shape where moments before there was only a glob. All the while, their feet work steadily, finding the right rhythm to keep everything moving. Sometimes, their lips move just a little, as though they were whispering to the clay the way one might whisper to a lover or share secrets with a friend- they seem to coax out beauty and share instructions with the clay, until it bends softly and forms the way they want it to. I was never any good at the pottery wheel, but  what I witnessed as other people worked was how I felt when I wrote in English class. I could grope around whisper to an idea and then keep my hands and pen moving rhythmically along until some essay appeared, fully formed on my page. It was was natural.

Then I started taking Spanish. Spanish did not come naturally to me at all. I remember this blind feeling of knowing what I wanted to say, but needing to look up not only the words, but the tense and the grammar and the punctuation. I’d get little phrases slightly off, I’d forget an accent mark or make a masculine verb feminine and then the whole sentence sounded wrong.  My teacher would carefully correct them, handing back a red stained piece of paper for me to go over again. I’d pull out the dictionary and go over each word, waiting for the moment when the words could flow effortlessly out of me and be magically correct on their own. When a phrase or sentence would “sound wrong” to me the way English does when there is a misused word.

Now, I am trying to learn to program. As I slog through lines of ruby, I get that same feeling I had in Spanish class. Learning to program is, of course, a foreign language. But it is also a language with a different set of rules.    I am not just trying to turn my english verbs and nouns into Spanish ones, I am trying to learn the very structure and sound of verbs that computers understand. Flowing descriptive sentences are chopped and spliced- removed of ambiguity and flourish and left as a series of commands strung together with a punctuation that is not dictated by the pauses and breaths of human speech. Again and again, my rake commands are greeted with lines of red errors detailing my every mistake. I grab a book and settle in, going over the code word by word one line at a time, and feel somehow super human (if only for a moment) when I clear an error. Then, when I get stuck, I show someone who actually knows what they are doing, and my super-human feeling completely vanishes as I watch them lay their fingers gently on the keyboard. They see instantly what “sounds wrong” and know exactly where to place the punctuation, when adjust a word or a space and how to clear out the red errors. And in the middle of it, some times I see their lips moving just slightly, coaxing the code to bend in just the right way.

I have a long way to go and a lot to learn. But with each error I start to recognize and each problem I figure out, I catch just a glimpse of that rhythm. One day when it, too, is natural, perhaps I will argue in praise of the art of  Ruby’s simplicity, its ease. For now, I fumble in the dark, waiting for the moment I can intuitively “hear” or feel how it is supposed to be and perhaps, even to whisper words of encouragement as the code takes shape.