Archive for the ‘tools’ Category

Sauce Labs

We got set up with Sauce Labs last week and are starting to play around with it. It seems like it will be a great addition to our automated testing. We are in the process of redefining how we use Selenium here, so that it is used more effectively and strategically.

Really, our greatest challenge has been getting everything smoothly integrated through Hudson, which I think will make a huge difference. The whole team worked through it together and we learned a ton.

At Agile 2010 I heard Jason Huggins speak about using selenium to make demos of functionality for shareholders. I loved this idea. What a great plan  to give people an easy way to keep track of a project’s current status and to nail down the functionality for some high level tests. We are trying a similar idea now for a widely used site to help users get a good handle on the features. We are hoping it is our first finalized example of the new automation process with sauce labs integrated. I am really excited!


Selenium IDE breaks firefox update 3.6.6

The new firefox update won’t start if the Selenium IDE is enabled.

OSX 10.6 (snow leopard)
firefox 3.6.6
Selenium IDE

I updated my firefox version, then tried to start the browser. nothing happened. I opened firefox in safemode with all add ons disabled. It started fine. I added my add ons back one at a time until I added selenium IDE again, and it couldn’t start after that.

I also went into about:config and added a new Boolean  setting: extensions.checkCompatibility.3.6b (false) as recommended here. Nothing seems to be working so far.

Has any one else had this problem yet?

Extending Lighthouse bug tracking

We use lighthouse for bug tracking at my company. It is simple, flexible and has a great api that allows us to hook it into testlink to track bugs from our test management system. The search functionality, however, seems limited. I have often wished there were a dashboard feature that allowed for more of an overview. It might be different if I worked on only one project, but as a QA lead,  I care about all the projects all the time. So when my company decided to do a hackathon day, I wanted to find a way to extend lighthouse into a dashboard.

We had 8 hours and 4 people to complete our hackathon task. Since a number of us had just completed a basic intro to Rails class, we decided to make a separate page that pulls data from Lighthouse. We were able to get the info out of lighthouse pretty easily with the API and Rails (with web themes) let us get a basic page up quickly. With so much data coming in, we had to figure out how to paginate the results so that the system didn’t run into any lag while trying to display the results. And the profile creation and filtering process took us a while.

By the end of the day, we had a working profile up with dynamic info coming in from our Lighthouse projects. A user can log in and see a list of their current tickets for all projects, with priority and status displayed and link those back to lighthouse to update them, or they can go to a more project specific view and see tasks assigned to them – with details for each project.

We have more work to do- we want to work on the sorting so that tickets appear in different bins by status, by priority, date updated etc. We also want to make a “team view” so that leads can log in and see the release status for all the members of their team. Eventually, we hope to do similar integration with testlink to display test status, test cases assigned and test cases created by you. We also want to integrate with Pivotal Tracker, which is also great from the project perspective, but needs a cross- project dashboard for those of us who are always working cross- team.

Lay Siege to your Servers

I had never used an open source load testing tool before I started working at my current company. But we are all about open source tools here, and have some great ones floating around. Siege was recommended by our Sys Admin Lead, who said he had not been able to find any other open source tools as powerful or easy to use as siege. With such a good recommendation, I set in to learn the program.

Siege is a load testing and benchmarking tool for web developers. It offers flexible options for customizing your script and then lays “siege” to your servers, testing their load capacity. It is incredibly handy, decently easy to use, and runs great on Linux  or OSX.

You can configure siege to test any number of concurrent users for a set amount of time.  Your tests can run against a single url, or to call a txt file containing a list of urls, so you can thoroughly test even complex webpages. You can download add ons like SPROXY from the same site that allows you to collect url data.

My favorite customizations are: -i (internet) to simulate internet users and -b (benchmark) which give s no delay for time testing. You can set a single test to repeat over again (-r) or print notifications to the screen (-v). You can even adjust the time delay (-d) to simulate actual human activity on the web page.

EX: here is a simple command, testing a single url.
 siege -c10 -t1 -i

Your script runs in the background (I run it on a different server, both for speed/accuracy and to keep my machine free) and when it finishes, it gives you your results and an overview of the test. Availability is the main thing to look at. It should always be at 100%, anything less means that users will have trouble accessing your site. Concurrency tells us the average number of simultaneous connections. Transaction rate is how fast each transaction was processed. These factors can all give you a good idea of what your website can handle.

Lifting the server siege...      done.
 Transactions:                 319 hits
 Availability:              100.00 %
 Elapsed time:               59.45 secs
 Data transferred:            0.75 MB
 Response time:                1.42 secs
 Transaction rate:            5.37 trans/sec
 Throughput:                0.01 MB/sec
 Concurrency:                7.63
 Successful transactions:         319
 Failed transactions:               0
 Longest transaction:           20.55
 Shortest transaction:            0.26

It is defiantly worth trying out to see if it fits your project, it has been a great addition to my QA toolbox.

Find a complete users manual and a download of the program on their main page: Joe Dog Software

QA toolbox- Firefox plug ins

Browser plug ins are invaluable tools for QA departments dealing with web software. Here are some very popular ones (and my favorites)…

Web Developer

I love this amazing, free testing tool. The Web Developer toolbar makes validations a breeze. You can perform CSS, HTML and link validations right from your menu bar.

HTMLL Validation with Web Developer

CSS Validation with Web Developer

It allows you to outline different page elements and display their title- which is really helpful when creating Selenium test scripts. You can also view your page without cookies or at a different resolution. This is one of the most helpful tools out there and I won’t sign off on a release with out it.


I use this mostly to track cookies, but it also offers debugging, element inspection, and error  tracking for Javascript and xml.

Firebug error tracking


Yslow is awesome. It works within the firebug window. Click over to the Yslow tab, open up the menu, and it grades the page you are on for performance. It even offers suggestions on how to make it faster.

Yslow- performance grades

DiffMerge for Text Update Testing

I just started using a Diffmerge for testing text updates to websites. It allows a graphical comparison of two text files and it highlights the differences between them. This is proving incredibly useful for big text updates to the websites I test. All you do is save a copy of the specification document with the new wording and then save a notepad of the text that appears on the site in staging. It will do the careful checking for you and clue you in to any inconsistencies.

If you have a lot of text updates, especially if they are legal notes or terms of service, it can be hard to catch every little error no matter how careful you are. This tool is easy and helpful and it gets you back to the functionality areas of testing quickly.

New CubicTest and Firefox?

I love experimenting with new testing tools and CubicTest’s selenium plug in for eclipse has kept coming out in my readings lately. So this week, I decided to give CubicTest a try. I already use Selenium and I’ve been hopeful about Cubic Test adding value to that tool. I downloaded the program and wrote a simple test to export via Selenium – anxious to see if we could adapt our current tests to the new tool. It didn’t take long for the first glaring red error to occur- firefox wasn’t even launched.

As it turns out, this seems to be an issue for the latest release of CubicTest- as I discovered from their forums:

I’ve just installed Cubictest using the update site on my Mac OSx.

I’ve just created on simple test which launches a website. When I click run, I get the error message ‘firefox-bin quit unexpectedly’. I also get a subsequent error message about

‘Check that

– The browser is installed (if in non-default dir, set it in PATH)

– The initial URL is correct

– There are no background (non-visible) browser processes hanging’

After discovering my issue echoed on the forums, I set to work looking for fixes. I ran through the whole battery of debug options. I tried different browsers, changed PATH and directory and adjusted settings. It seems utterly unable to export selenium tests.

I also came across this fix, which finally shed more light on the issue -but even with the steps suggested here, the issue still appears.

The issue is mostly related to libsqlite3.dylib that resides on the Mac OSx.

When Firefox starts, it modifies the system’s DYLD_LIBRARY_PATH to point it to its own, this allows FF to run with the latest versions of all its libraries. However as you’ve seen in Eclispe, when Selenium starts up, it forces FF to use the system’s DYLD_LIBRARY_PATH (thus the system’s libraries), causing it the crash.

The solution is to prevent that and make Selenium/CubicTest plugin launch FF without the above. There are a couple of steps to get there:

  1. Create a bash-script in your user directory. Call it firefox-bin (the same name as that used by firefox). This is not really necessary but it helps keep things tidy and neat.
  2. Add the following content to the script file:
    • #!/bin/sh
    • /Applications/ $@
    • exit $?
  3. Next, edit eclipse.ini to use:
    • -DfirefoxDefaultPath=/Users/<username>/Scripts/firefox-bin
  4. Restart Eclipse and try to run your test. It should launch FF fine. The first instance launched will not run the test (not sure why yet). AppleCMD+Q to quit. It will by itself launch a second instance after you’ve quit the first one, that instance would run your test / tests.

The program looks really interesting. Selenium has worked well on our project before- and extending it capibilities and maintainablility is an exciting prospect. I’ll keep checking back to see if this issue is resolved.

Has anyone else had luck with this program?

I may give it another try on a windows virtual box -but with less than exciting results so far, I may keep an eye on this issue to see if it is resolved first.