Posts Tagged ‘Automated Testing’

Sauce Labs

We got set up with Sauce Labs last week and are starting to play around with it. It seems like it will be a great addition to our automated testing. We are in the process of redefining how we use Selenium here, so that it is used more effectively and strategically.

Really, our greatest challenge has been getting everything smoothly integrated through Hudson, which I think will make a huge difference. The whole team worked through it together and we learned a ton.

At Agile 2010 I heard Jason Huggins speak about using selenium to make demos of functionality for shareholders. I loved this idea. What a great plan  to give people an easy way to keep track of a project’s current status and to nail down the functionality for some high level tests. We are trying a similar idea now for a widely used site to help users get a good handle on the features. We are hoping it is our first finalized example of the new automation process with sauce labs integrated. I am really excited!

Advertisements

New CubicTest and Firefox?

I love experimenting with new testing tools and CubicTest’s selenium plug in for eclipse has kept coming out in my readings lately. So this week, I decided to give CubicTest a try. I already use Selenium and I’ve been hopeful about Cubic Test adding value to that tool. I downloaded the program and wrote a simple test to export via Selenium – anxious to see if we could adapt our current tests to the new tool. It didn’t take long for the first glaring red error to occur- firefox wasn’t even launched.

As it turns out, this seems to be an issue for the latest release of CubicTest- as I discovered from their forums:

I’ve just installed Cubictest using the update site on my Mac OSx.

I’ve just created on simple test which launches a website. When I click run, I get the error message ‘firefox-bin quit unexpectedly’. I also get a subsequent error message about

‘Check that

– The browser is installed (if in non-default dir, set it in PATH)

– The initial URL is correct

– There are no background (non-visible) browser processes hanging’

After discovering my issue echoed on the forums, I set to work looking for fixes. I ran through the whole battery of debug options. I tried different browsers, changed PATH and directory and adjusted settings. It seems utterly unable to export selenium tests.

I also came across this fix, which finally shed more light on the issue -but even with the steps suggested here, the issue still appears.

The issue is mostly related to libsqlite3.dylib that resides on the Mac OSx.

When Firefox starts, it modifies the system’s DYLD_LIBRARY_PATH to point it to its own, this allows FF to run with the latest versions of all its libraries. However as you’ve seen in Eclispe, when Selenium starts up, it forces FF to use the system’s DYLD_LIBRARY_PATH (thus the system’s libraries), causing it the crash.

The solution is to prevent that and make Selenium/CubicTest plugin launch FF without the above. There are a couple of steps to get there:

  1. Create a bash-script in your user directory. Call it firefox-bin (the same name as that used by firefox). This is not really necessary but it helps keep things tidy and neat.
  2. Add the following content to the script file:
    • #!/bin/sh
    • /Applications/Firefox.app/Contents/MacOS/firefox-bin $@
    • exit $?
  3. Next, edit eclipse.ini to use:
    • -DfirefoxDefaultPath=/Users/<username>/Scripts/firefox-bin
  4. Restart Eclipse and try to run your test. It should launch FF fine. The first instance launched will not run the test (not sure why yet). AppleCMD+Q to quit. It will by itself launch a second instance after you’ve quit the first one, that instance would run your test / tests.

The program looks really interesting. Selenium has worked well on our project before- and extending it capibilities and maintainablility is an exciting prospect. I’ll keep checking back to see if this issue is resolved.

Has anyone else had luck with this program?

I may give it another try on a windows virtual box -but with less than exciting results so far, I may keep an eye on this issue to see if it is resolved first.

Litmus Review

I’ve been using litmus app for about four months now. It allows you to view a web page on many different browsers/OS combination very easily. It also has some support for email viewing in different clients. Litmus helps to detect UI issues on configurations you don’t directly use for testing.

The Good:

support for email UI is nice- but limited

You can be notified via IM or email when your tests are ready

individual test cases can be rerun without waiting for all to execute again

Simple and straightforward

lots of browsers available for testing

The Bad:

no way to chose which version of the operating system you want to test on

intermittent errors seem to occur because it doesnt wait for pages to load (even when you have selected allow extra time)

emails sent automatically via a 3rd party cannot be checked, as you have to directly send an email to the address they provide.

overall:

It is a good sanity check and reference for UI issues. It supports a lot of browsers and is very easy to use. Main browsers should defiantly still be checked manually, and Litmus needs to be reviewed (as with any automated test result) with a critical eye for tool failures. In all, a good app and an asset to the QA process.