Archive for the ‘Review’ Category

Extending Lighthouse bug tracking

We use lighthouse for bug tracking at my company. It is simple, flexible and has a great api that allows us to hook it into testlink to track bugs from our test management system. The search functionality, however, seems limited. I have often wished there were a dashboard feature that allowed for more of an overview. It might be different if I worked on only one project, but as a QA lead,  I care about all the projects all the time. So when my company decided to do a hackathon day, I wanted to find a way to extend lighthouse into a dashboard.

We had 8 hours and 4 people to complete our hackathon task. Since a number of us had just completed a basic intro to Rails class, we decided to make a separate page that pulls data from Lighthouse. We were able to get the info out of lighthouse pretty easily with the API and Rails (with web themes) let us get a basic page up quickly. With so much data coming in, we had to figure out how to paginate the results so that the system didn’t run into any lag while trying to display the results. And the profile creation and filtering process took us a while.

By the end of the day, we had a working profile up with dynamic info coming in from our Lighthouse projects. A user can log in and see a list of their current tickets for all projects, with priority and status displayed and link those back to lighthouse to update them, or they can go to a more project specific view and see tasks assigned to them – with details for each project.

We have more work to do- we want to work on the sorting so that tickets appear in different bins by status, by priority, date updated etc. We also want to make a “team view” so that leads can log in and see the release status for all the members of their team. Eventually, we hope to do similar integration with testlink to display test status, test cases assigned and test cases created by you. We also want to integrate with Pivotal Tracker, which is also great from the project perspective, but needs a cross- project dashboard for those of us who are always working cross- team.


The “Mythical” QA schedule

I just finished reading The Mythical Man Month by Frederick Brooks. This classic was still amazingly relevant, even thirty some years later.

Some of the tactics he suggests have just been made obsolete by the advances in technology – however, his use of a surgical team as a model for software teams seemed surprisingly agile and modern (not that agile and modern are synonymous). What was most interesting to me was the importance he placed on software testing. In his planning for software schedules, he allots half the project time to testing and bug fixes.

What could you do with that kind of time? Could better fixes be implemented for bugs? Would more bugs be fixed pre-launch? I find that as development schedules get pushed out, it is testing that get shortened.

Squished between milestones that can’t move or running up against a launch date that has already been announced, testing is often a race against time. Sometimes – new projects are hard to estimate for. Sure, you know how long it will take you to run through your test cases, but what about the debugging time? That varies greatly depending on the bugs found. Having half the schedule devoted to testing (or even just a greater percentage) would certainly be a change.

Would such a schedule just stretch things out or get shortened itself? Would it hinder the creativity of the engineers by shortening their time? Or would it work to allow a team with greater importance and visibility placed on testing? A place where quality was not just a “QA” thing, but a collaborative time of testing and bug fixing that could bring the importance of quality at every step of the development process to the forefront.

Lay Siege to your Servers

I had never used an open source load testing tool before I started working at my current company. But we are all about open source tools here, and have some great ones floating around. Siege was recommended by our Sys Admin Lead, who said he had not been able to find any other open source tools as powerful or easy to use as siege. With such a good recommendation, I set in to learn the program.

Siege is a load testing and benchmarking tool for web developers. It offers flexible options for customizing your script and then lays “siege” to your servers, testing their load capacity. It is incredibly handy, decently easy to use, and runs great on Linux  or OSX.

You can configure siege to test any number of concurrent users for a set amount of time.  Your tests can run against a single url, or to call a txt file containing a list of urls, so you can thoroughly test even complex webpages. You can download add ons like SPROXY from the same site that allows you to collect url data.

My favorite customizations are: -i (internet) to simulate internet users and -b (benchmark) which give s no delay for time testing. You can set a single test to repeat over again (-r) or print notifications to the screen (-v). You can even adjust the time delay (-d) to simulate actual human activity on the web page.

EX: here is a simple command, testing a single url.
 siege -c10 -t1 -i

Your script runs in the background (I run it on a different server, both for speed/accuracy and to keep my machine free) and when it finishes, it gives you your results and an overview of the test. Availability is the main thing to look at. It should always be at 100%, anything less means that users will have trouble accessing your site. Concurrency tells us the average number of simultaneous connections. Transaction rate is how fast each transaction was processed. These factors can all give you a good idea of what your website can handle.

Lifting the server siege...      done.
 Transactions:                 319 hits
 Availability:              100.00 %
 Elapsed time:               59.45 secs
 Data transferred:            0.75 MB
 Response time:                1.42 secs
 Transaction rate:            5.37 trans/sec
 Throughput:                0.01 MB/sec
 Concurrency:                7.63
 Successful transactions:         319
 Failed transactions:               0
 Longest transaction:           20.55
 Shortest transaction:            0.26

It is defiantly worth trying out to see if it fits your project, it has been a great addition to my QA toolbox.

Find a complete users manual and a download of the program on their main page: Joe Dog Software

QA toolbox- Firefox plug ins

Browser plug ins are invaluable tools for QA departments dealing with web software. Here are some very popular ones (and my favorites)…

Web Developer

I love this amazing, free testing tool. The Web Developer toolbar makes validations a breeze. You can perform CSS, HTML and link validations right from your menu bar.

HTMLL Validation with Web Developer

CSS Validation with Web Developer

It allows you to outline different page elements and display their title- which is really helpful when creating Selenium test scripts. You can also view your page without cookies or at a different resolution. This is one of the most helpful tools out there and I won’t sign off on a release with out it.


I use this mostly to track cookies, but it also offers debugging, element inspection, and error  tracking for Javascript and xml.

Firebug error tracking


Yslow is awesome. It works within the firebug window. Click over to the Yslow tab, open up the menu, and it grades the page you are on for performance. It even offers suggestions on how to make it faster.

Yslow- performance grades

New CubicTest and Firefox?

I love experimenting with new testing tools and CubicTest’s selenium plug in for eclipse has kept coming out in my readings lately. So this week, I decided to give CubicTest a try. I already use Selenium and I’ve been hopeful about Cubic Test adding value to that tool. I downloaded the program and wrote a simple test to export via Selenium – anxious to see if we could adapt our current tests to the new tool. It didn’t take long for the first glaring red error to occur- firefox wasn’t even launched.

As it turns out, this seems to be an issue for the latest release of CubicTest- as I discovered from their forums:

I’ve just installed Cubictest using the update site on my Mac OSx.

I’ve just created on simple test which launches a website. When I click run, I get the error message ‘firefox-bin quit unexpectedly’. I also get a subsequent error message about

‘Check that

– The browser is installed (if in non-default dir, set it in PATH)

– The initial URL is correct

– There are no background (non-visible) browser processes hanging’

After discovering my issue echoed on the forums, I set to work looking for fixes. I ran through the whole battery of debug options. I tried different browsers, changed PATH and directory and adjusted settings. It seems utterly unable to export selenium tests.

I also came across this fix, which finally shed more light on the issue -but even with the steps suggested here, the issue still appears.

The issue is mostly related to libsqlite3.dylib that resides on the Mac OSx.

When Firefox starts, it modifies the system’s DYLD_LIBRARY_PATH to point it to its own, this allows FF to run with the latest versions of all its libraries. However as you’ve seen in Eclispe, when Selenium starts up, it forces FF to use the system’s DYLD_LIBRARY_PATH (thus the system’s libraries), causing it the crash.

The solution is to prevent that and make Selenium/CubicTest plugin launch FF without the above. There are a couple of steps to get there:

  1. Create a bash-script in your user directory. Call it firefox-bin (the same name as that used by firefox). This is not really necessary but it helps keep things tidy and neat.
  2. Add the following content to the script file:
    • #!/bin/sh
    • /Applications/ $@
    • exit $?
  3. Next, edit eclipse.ini to use:
    • -DfirefoxDefaultPath=/Users/<username>/Scripts/firefox-bin
  4. Restart Eclipse and try to run your test. It should launch FF fine. The first instance launched will not run the test (not sure why yet). AppleCMD+Q to quit. It will by itself launch a second instance after you’ve quit the first one, that instance would run your test / tests.

The program looks really interesting. Selenium has worked well on our project before- and extending it capibilities and maintainablility is an exciting prospect. I’ll keep checking back to see if this issue is resolved.

Has anyone else had luck with this program?

I may give it another try on a windows virtual box -but with less than exciting results so far, I may keep an eye on this issue to see if it is resolved first.

Litmus Review

I’ve been using litmus app for about four months now. It allows you to view a web page on many different browsers/OS combination very easily. It also has some support for email viewing in different clients. Litmus helps to detect UI issues on configurations you don’t directly use for testing.

The Good:

support for email UI is nice- but limited

You can be notified via IM or email when your tests are ready

individual test cases can be rerun without waiting for all to execute again

Simple and straightforward

lots of browsers available for testing

The Bad:

no way to chose which version of the operating system you want to test on

intermittent errors seem to occur because it doesnt wait for pages to load (even when you have selected allow extra time)

emails sent automatically via a 3rd party cannot be checked, as you have to directly send an email to the address they provide.


It is a good sanity check and reference for UI issues. It supports a lot of browsers and is very easy to use. Main browsers should defiantly still be checked manually, and Litmus needs to be reviewed (as with any automated test result) with a critical eye for tool failures. In all, a good app and an asset to the QA process.

Test Link Review


If you search for test tracking and management software, you will run into Test link. Testlink, though far from perfect, does some pretty cool things. I love any excuse to try out a cool new program (Which is the best part about working in a tech environment, isn’t it?) so I decided to give it a shot.  I used to keep track of my tests on proprietary software and open office spreadsheets, but the organization offered by testlink moved my whole testing approach to a new level.

The Good:



requirements management

requirements based testing

result reporting

central location for assigning, approving and executing tests

Open Source and written in PHP, so it can be adjusted for your specific project

The Bad:

It is a bit clunky to use

Minor bugs and unfinished features

Isn’t set up for a ton of integration- but there is an API

The results:

Testing process has been opened up for anyone on the project to be aware of what QA is working on. Communication between QA and Devs has improved, and the approval process is much easier and more organic than holding a big meeting. Test link has the potential for seamless integration with bug filing and requirements management, helping to carefully order and maintain a tester’s world. It doesn’t come pre-linked to much, but the site does offer some instruction for that. It does offer a good user guide and a pretty active community for support.

You can input your documents (I haven’t found a way to do this automatically, but manually putting them in does ensure that you do careful ambiguity review) and then link each requirement to a test case, or generate test cases right from the requirement. The execute section can accept xml results and notes. Testlink automatically generates test plans, test reports with data from the system, so you can put out a document that shows your test plan, and then gives overviews of your test cases and tracks the requriements. It has been extremely useful for increasing transparency in the QA department and keeping everyone involved with the process. There is a way to create custom fields that I have been using for test plan approvals and linking to bugs filed.

In all, it is a great program- that could use some finishing tweaks and an UI overhaul. I can look past its flaws though for the functionality it provides. I’ll admit it- I love testlink.

Also read this post here: