Archive for the ‘Automated Testing’ Category

Selenium IDE breaks firefox update 3.6.6

The new firefox update won’t start if the Selenium IDE is enabled.

OSX 10.6 (snow leopard)
firefox 3.6.6
Selenium IDE

I updated my firefox version, then tried to start the browser. nothing happened. I opened firefox in safemode with all add ons disabled. It started fine. I added my add ons back one at a time until I added selenium IDE again, and it couldn’t start after that.

I also went into about:config and added a new Boolean  setting: extensions.checkCompatibility.3.6b (false) as recommended here. Nothing seems to be working so far.

Has any one else had this problem yet?


Lay Siege to your Servers

I had never used an open source load testing tool before I started working at my current company. But we are all about open source tools here, and have some great ones floating around. Siege was recommended by our Sys Admin Lead, who said he had not been able to find any other open source tools as powerful or easy to use as siege. With such a good recommendation, I set in to learn the program.

Siege is a load testing and benchmarking tool for web developers. It offers flexible options for customizing your script and then lays “siege” to your servers, testing their load capacity. It is incredibly handy, decently easy to use, and runs great on Linux  or OSX.

You can configure siege to test any number of concurrent users for a set amount of time.  Your tests can run against a single url, or to call a txt file containing a list of urls, so you can thoroughly test even complex webpages. You can download add ons like SPROXY from the same site that allows you to collect url data.

My favorite customizations are: -i (internet) to simulate internet users and -b (benchmark) which give s no delay for time testing. You can set a single test to repeat over again (-r) or print notifications to the screen (-v). You can even adjust the time delay (-d) to simulate actual human activity on the web page.

EX: here is a simple command, testing a single url.
 siege -c10 -t1 -i

Your script runs in the background (I run it on a different server, both for speed/accuracy and to keep my machine free) and when it finishes, it gives you your results and an overview of the test. Availability is the main thing to look at. It should always be at 100%, anything less means that users will have trouble accessing your site. Concurrency tells us the average number of simultaneous connections. Transaction rate is how fast each transaction was processed. These factors can all give you a good idea of what your website can handle.

Lifting the server siege...      done.
 Transactions:                 319 hits
 Availability:              100.00 %
 Elapsed time:               59.45 secs
 Data transferred:            0.75 MB
 Response time:                1.42 secs
 Transaction rate:            5.37 trans/sec
 Throughput:                0.01 MB/sec
 Concurrency:                7.63
 Successful transactions:         319
 Failed transactions:               0
 Longest transaction:           20.55
 Shortest transaction:            0.26

It is defiantly worth trying out to see if it fits your project, it has been a great addition to my QA toolbox.

Find a complete users manual and a download of the program on their main page: Joe Dog Software

Documentation on Agile Developement Teams

Agile Teams are all about flexibility, right? Rules evolve with the project, people’s suggestions can taken on, deadlines can be adjusted internally – and within that one week sprint, the team is pretty much free to set their own pace and schedule. Its great. It allows for creativity, changes in plans and new ideas. But sometimes, it seems like documents lose some importance.

I know, I know. People get tired of hearing QA whine about documentation. But there is a reason.  In a world where you are free to find your own way, it is good to have some idea where you are finding your way too – so that you- and your teammates – all end up at the same destination. Documents provide that overview, that roadmap. They also help others to follow you- either when making revisions in the future, or when testing the final output. No matter which development style your team follows, the importance of good documentation should not be lost.

In waterfall development, documents are turn by turn directions -carefully planned and harder to change. Agile developement needs the same directions, but the exact path can be filled in as you go. By the end of the project (by the time QA starts testing), Agile teams should sh0uld have produced the same kind of documentation that waterfall teams have.

But what about QA teams? How do you go about designing your test plan when you don’t know exactly how things are going to work yet? The documents won’t be ready in advance, which means you have to keep your plan flexible… agile. You should have a good idea of the final product, so you can plan your approach, your general tests, your schedule etc very early on. As documents are updated, update the test plan. This was a big adjustment for me, as I was used to knowing in advance what all will be done. But really, the philosophy is the same- you just work in phases right behind the developers. They finish a piece of functionality, you finish your test plan and test that piece while they start filling in the documents on the next piece. QA defiantly has an important place here – and keeping an eye on the requirements, performing ambiguity review at each stage and applying those to an agile test plan are all part of it.

Because with out requirements, development really is like that old elephant story. (An awesome version of which is reposted below from this website)

It was six men of Indostan
To learning much inclined,
Who went to see the Elephant
(Though all of them were blind),
That each by observation
Might satisfy his mind.

The First approach’d the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
“God bless me! but the Elephant
Is very like a wall!”

The Second, feeling of the tusk,
Cried, -“Ho! what have we here
So very round and smooth and sharp?
To me ’tis mighty clear
This wonder of an Elephant
Is very like a spear!”

The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up and spake:
“I see,” quoth he, “the Elephant
Is very like a snake!”

The Fourth reached out his eager hand,
And felt about the knee.
“What most this wondrous beast is like
Is mighty plain,” quoth he,
“‘Tis clear enough the Elephant
Is very like a tree!”

The Fifth, who chanced to touch the ear,
Said: “E’en the blindest man
Can tell what this resembles most;
Deny the fact who can,
This marvel of an Elephant
Is very like a fan!”

The Sixth no sooner had begun
About the beast to grope,
Then, seizing on the swinging tail
That fell within his scope,
“I see,” quoth he, “the Elephant
Is very like a rope!”

And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!


So oft in theologic wars,
The disputants, I ween,
Rail on in utter ignorance
Of what each other mean,
And prate about an Elephant
Not one of them has seen!

Litmus Review

I’ve been using litmus app for about four months now. It allows you to view a web page on many different browsers/OS combination very easily. It also has some support for email viewing in different clients. Litmus helps to detect UI issues on configurations you don’t directly use for testing.

The Good:

support for email UI is nice- but limited

You can be notified via IM or email when your tests are ready

individual test cases can be rerun without waiting for all to execute again

Simple and straightforward

lots of browsers available for testing

The Bad:

no way to chose which version of the operating system you want to test on

intermittent errors seem to occur because it doesnt wait for pages to load (even when you have selected allow extra time)

emails sent automatically via a 3rd party cannot be checked, as you have to directly send an email to the address they provide.


It is a good sanity check and reference for UI issues. It supports a lot of browsers and is very easy to use. Main browsers should defiantly still be checked manually, and Litmus needs to be reviewed (as with any automated test result) with a critical eye for tool failures. In all, a good app and an asset to the QA process.