Archive for the ‘testing toolbox’ Category

Sauce Labs

We got set up with Sauce Labs last week and are starting to play around with it. It seems like it will be a great addition to our automated testing. We are in the process of redefining how we use Selenium here, so that it is used more effectively and strategically.

Really, our greatest challenge has been getting everything smoothly integrated through Hudson, which I think will make a huge difference. The whole team worked through it together and we learned a ton.

At Agile 2010 I heard Jason Huggins speak about using selenium to make demos of functionality for shareholders. I loved this idea. What a great plan  to give people an easy way to keep track of a project’s current status and to nail down the functionality for some high level tests. We are trying a similar idea now for a widely used site to help users get a good handle on the features. We are hoping it is our first finalized example of the new automation process with sauce labs integrated. I am really excited!

Advertisements

Selenium IDE breaks firefox update 3.6.6

The new firefox update won’t start if the Selenium IDE is enabled.

System:
OSX 10.6 (snow leopard)
firefox 3.6.6
Selenium IDE

I updated my firefox version, then tried to start the browser. nothing happened. I opened firefox in safemode with all add ons disabled. It started fine. I added my add ons back one at a time until I added selenium IDE again, and it couldn’t start after that.

I also went into about:config and added a new Boolean  setting: extensions.checkCompatibility.3.6b (false) as recommended here. Nothing seems to be working so far.

Has any one else had this problem yet?

Extending Lighthouse bug tracking

We use lighthouse for bug tracking at my company. It is simple, flexible and has a great api that allows us to hook it into testlink to track bugs from our test management system. The search functionality, however, seems limited. I have often wished there were a dashboard feature that allowed for more of an overview. It might be different if I worked on only one project, but as a QA lead,  I care about all the projects all the time. So when my company decided to do a hackathon day, I wanted to find a way to extend lighthouse into a dashboard.

We had 8 hours and 4 people to complete our hackathon task. Since a number of us had just completed a basic intro to Rails class, we decided to make a separate page that pulls data from Lighthouse. We were able to get the info out of lighthouse pretty easily with the API and Rails (with web themes) let us get a basic page up quickly. With so much data coming in, we had to figure out how to paginate the results so that the system didn’t run into any lag while trying to display the results. And the profile creation and filtering process took us a while.

By the end of the day, we had a working profile up with dynamic info coming in from our Lighthouse projects. A user can log in and see a list of their current tickets for all projects, with priority and status displayed and link those back to lighthouse to update them, or they can go to a more project specific view and see tasks assigned to them – with details for each project.

We have more work to do- we want to work on the sorting so that tickets appear in different bins by status, by priority, date updated etc. We also want to make a “team view” so that leads can log in and see the release status for all the members of their team. Eventually, we hope to do similar integration with testlink to display test status, test cases assigned and test cases created by you. We also want to integrate with Pivotal Tracker, which is also great from the project perspective, but needs a cross- project dashboard for those of us who are always working cross- team.

Lay Siege to your Servers

I had never used an open source load testing tool before I started working at my current company. But we are all about open source tools here, and have some great ones floating around. Siege was recommended by our Sys Admin Lead, who said he had not been able to find any other open source tools as powerful or easy to use as siege. With such a good recommendation, I set in to learn the program.

Siege is a load testing and benchmarking tool for web developers. It offers flexible options for customizing your script and then lays “siege” to your servers, testing their load capacity. It is incredibly handy, decently easy to use, and runs great on Linux  or OSX.

You can configure siege to test any number of concurrent users for a set amount of time.  Your tests can run against a single url, or to call a txt file containing a list of urls, so you can thoroughly test even complex webpages. You can download add ons like SPROXY from the same site that allows you to collect url data.

My favorite customizations are: -i (internet) to simulate internet users and -b (benchmark) which give s no delay for time testing. You can set a single test to repeat over again (-r) or print notifications to the screen (-v). You can even adjust the time delay (-d) to simulate actual human activity on the web page.

EX: here is a simple command, testing a single url.
 siege -c10 -t1 -i http://www.joedog.org/index/siege-home


Your script runs in the background (I run it on a different server, both for speed/accuracy and to keep my machine free) and when it finishes, it gives you your results and an overview of the test. Availability is the main thing to look at. It should always be at 100%, anything less means that users will have trouble accessing your site. Concurrency tells us the average number of simultaneous connections. Transaction rate is how fast each transaction was processed. These factors can all give you a good idea of what your website can handle.

Lifting the server siege...      done.
 Transactions:                 319 hits
 Availability:              100.00 %
 Elapsed time:               59.45 secs
 Data transferred:            0.75 MB
 Response time:                1.42 secs
 Transaction rate:            5.37 trans/sec
 Throughput:                0.01 MB/sec
 Concurrency:                7.63
 Successful transactions:         319
 Failed transactions:               0
 Longest transaction:           20.55
 Shortest transaction:            0.26


It is defiantly worth trying out to see if it fits your project, it has been a great addition to my QA toolbox.

Find a complete users manual and a download of the program on their main page: Joe Dog Software

QA toolbox- Firefox plug ins

Browser plug ins are invaluable tools for QA departments dealing with web software. Here are some very popular ones (and my favorites)…

Web Developer

I love this amazing, free testing tool. The Web Developer toolbar makes validations a breeze. You can perform CSS, HTML and link validations right from your menu bar.

HTMLL Validation with Web Developer

CSS Validation with Web Developer

It allows you to outline different page elements and display their title- which is really helpful when creating Selenium test scripts. You can also view your page without cookies or at a different resolution. This is one of the most helpful tools out there and I won’t sign off on a release with out it.

Firebug

I use this mostly to track cookies, but it also offers debugging, element inspection, and error  tracking for Javascript and xml.

Firebug error tracking

Yslow

Yslow is awesome. It works within the firebug window. Click over to the Yslow tab, open up the menu, and it grades the page you are on for performance. It even offers suggestions on how to make it faster.

Yslow- performance grades

DiffMerge for Text Update Testing

I just started using a Diffmerge for testing text updates to websites. It allows a graphical comparison of two text files and it highlights the differences between them. This is proving incredibly useful for big text updates to the websites I test. All you do is save a copy of the specification document with the new wording and then save a notepad of the text that appears on the site in staging. It will do the careful checking for you and clue you in to any inconsistencies.

If you have a lot of text updates, especially if they are legal notes or terms of service, it can be hard to catch every little error no matter how careful you are. This tool is easy and helpful and it gets you back to the functionality areas of testing quickly.

Litmus Review

I’ve been using litmus app for about four months now. It allows you to view a web page on many different browsers/OS combination very easily. It also has some support for email viewing in different clients. Litmus helps to detect UI issues on configurations you don’t directly use for testing.

The Good:

support for email UI is nice- but limited

You can be notified via IM or email when your tests are ready

individual test cases can be rerun without waiting for all to execute again

Simple and straightforward

lots of browsers available for testing

The Bad:

no way to chose which version of the operating system you want to test on

intermittent errors seem to occur because it doesnt wait for pages to load (even when you have selected allow extra time)

emails sent automatically via a 3rd party cannot be checked, as you have to directly send an email to the address they provide.

overall:

It is a good sanity check and reference for UI issues. It supports a lot of browsers and is very easy to use. Main browsers should defiantly still be checked manually, and Litmus needs to be reviewed (as with any automated test result) with a critical eye for tool failures. In all, a good app and an asset to the QA process.