Experiments and reviews of various ui testing frameworks

mkdir ui-testing
cd ui-testing

echo '{ name:ui-testing}' > package.json

npm install --save-dev selenium-webdriver
npm install --save-dev browserstack-webdriver
npm install --save-dev wd
npm install --save-dev webdriverio
npm install --save-dev kommando
npm install --save-dev nightwatch
npm install --save-dev karma
npm install --save-dev leadfoot
npm install --save-dev intern

npm install --save-dev jscoverage
npm install --save-dev blanket

git remote add origin https://github.com/ainthek/ui-testing.git

echo node_modules > .gitignore

Texts come from their web sites

The official WebDriver JavaScript bindings from the Selenium project

• https://www.npmjs.com/package/selenium-webdriver
• https://code.google.com/p/selenium/

BrowserStack WebDriver JavaScript bindings with keep alive support

• https://www.npmjs.com/package/browserstack-webdriver

WebDriver/Selenium 2 node.js client

• https://www.npmjs.com/package/wd

Selenium 2.0 bindings for NodeJS

Most of the Selenium WebDriver Wire Protocol is already implemented and wrapped in useful commands.

Adding helper functions, or more complicated sets and combinations of existing commands is simple and really useful

WebdriverIO works in combination with most of the TDD and BDD test frameworks in the JavaScript world.

• https://www.npmjs.com/package/webdriverio
• http://webdriver.io

A node.js bindings implementation for selenium 2.0/webdriver

UI automated testing framework powered by Node.js. It uses the Selenium WebDriver API.

• https://www.npmjs.com/package/nightwatch

Configurable cross browser functional / acceptance test launcher (using Webdriver)

Spectacular Test Runner for JavaScript.

A simple tool that allows you to execute JavaScript code in multiple real browsers.

The main purpose of Karma is to make your TDD development easy, fast, and fun

Intern. A next-generation code testing stack for JavaScript.

Intern is a complete test stack for JavaScript designed to help you write and run consistent, high-quality test cases for your JavaScript libraries and applications. It can be used to test any JavaScript code.

Its functional testing capabilities can even be used to test non-JavaScript Web and mobile apps, if you really want.

• http://theintern.io
• https://www.npmjs.com/package/intern

A seamless JavaScript code coverage library.

• https://www.npmjs.com/package/blanket

a javascript coverage tool, can be used in node dev, and browser side js dev

• https://www.npmjs.com/package/jscoverage
• Quick Windows (MINGW) Fix

generated Mon Feb 16 09:52:00 CEST 2015

Included here is not comparable, some of them are full stack things some of them are modules reused by others etc...

However the numbers tell something (TODO: separate and chart)

Far from being finished....

Intern web site contains some comparisons of the frameworks, however it is missing some important players and features. So I will add some here:

Subjective Rankings:

3-good
2-ok
1-poor
0-none

How much manual work is needed to run tests against browsers that require selenium-server-standalone-*.jar. Download, configure and instantiate support in each 'framework'.

• (cons) 'To run the tests, you will need to download a copy of the ChromeDriver and make sure it can be found on your PATH.'

• (cons) They don't event updated docs (nor npm nor wiki) after implementation of direct Firefox support.

• However after some downloading of related and compatible version for each of your OSes, you can have it running in lt; 30 mins for all browsers you need.

  TODO: download sample code sample

• why using pretty old 'selenium-version': '2.41.0' in the example config ?
• based on this blog post you must download all manually as well

Actually 'installing' Selenium, Chromedriver, IEDriver and PhantomJS is not node by kommando, but by vebrdvr module.

to automate download of Standalone Server and Drivers

TODO:

Here I ry to evaluate how much time and browsing it took to start first functional test in local browser. Intern is in disadvantage here, their docs cover bit more so orientation is more difficult.

running sample from npm readme with native FF support they have now, took under 10 seconds, if you have FF installed.

Running REPL and starting browser (FF, chrome, Opera) right after install requires no reading of any docs. This is the main purpose of the kommando module (even if implemented using vebdrvr).

$ ./node_modules/.bin/kommando --runner repl --browser chrome
Starting Selenium server ...
Selenium server started at: http://192.168.x.x:49408/wd/hub
Run tests using "browserName=chrome"
"kommando" / "kommando.browser" provide access to the running browser instance
kommando>

After 30 minutes of quick searching, I have found this topic: Running-your-own-WebDriver-server. After another 30 minutes I have still no functional testing sample running. Another 30mins, cloned tutorial, downloaded full tutorial cannot make tests/functional to run, no errors, just not run.

Gave up, asking for help

Update: 2015/02/10 - reading the answer (Thank) and googling I have found this http://www.sitepen.com/blog/2014/04/23/dojo-faq-how-can-i-run-dojo-tests-locally-with-intern/ article. After not believing this is really what intern offers, I have 'managed' to start server MANULLLY with actually typing the word 'java' (GRRRRRR, just compare wit the bare selenium-webdriver, which 'they' do not consider to be a test framework , and after previous positive KARMA, seems unbelievable achievement..

After "pruning my environment" (as vaguely suggested in article), to my guessed minimum:

{ browserName: 'firefox' }

or

{ browserName: 'chrome' }

and having test-code:

define([
	'intern!object',
], function (registerSuite) {
     registerSuite({
        name: 'minimal demo',
        'of getting google': function () {
            return this.remote.get('http://www.google.com');
        }
    });
});	

and running:

(GRRRRRR) java -jar $SELENIUM_SERVER_JAR -port 4444

and

node_modules/.bin/intern-runner config=tests/intern

I have finally seen 'flash' of google.com webpage , after series of blinking URLs.... Success withing 48 hours in not bad. Or ? Going to sleep (today).

• support of coverage reporting using 5 years old tj's node-jscoverage
• tj claims in README.txt that node module jscoverage npm module DOES NOT WORK
• jscoverage npm module
  • it has 16158 downloads in the last month
  • and git gub repo seems more recent even if not active
  • readme covers instruction how to test
• tj's version does not uninstall on OSX 10.9.5
• a lot of outdated? infos on web, including this one http://seejohncode.com/2012/03/13/setting-up-mocha-jscoverage/

asking for help

Finally I have decided to ignore mochas docs and tj's repo and tried very simple one minute setup and example:

npm install --save-dev jscoverage

mocha -r jscoverage -R spec tests/mocha/test.js

Coverage result
  ✓ tests/mocha/test.js	line[100%]  branch[100%]
  ⁍ lib/module.js	line[ 66%]  branch[100%]

  1 passing (5ms)

After trying tests on my other Win (MINGW) machine, neeeded to quickly patch jscoverage.

And also received link from boneskull about blanket, so will try it.

Testing blanket on windows (MINGW) ends with no conevarge stats at all, quick look at the node+mocha detection reveals why:

if (args[0] === 'node' &&
    args[1].indexOf(join('node_modules','mocha','bin')) > -1 &&
    blanketRequired){

    //using mocha cli
    module.exports = blanketNode(null,true);

So probably I'm going again for Fork-reconfigure-package.json-Fix-Pull-Request-Hope cycle ? Same as with jscoverage ?

Not today ;-(