-
Notifications
You must be signed in to change notification settings - Fork 37
Writing integration tests
Yay, you're going to help us test our entire software stack to make sure all components work together? We want this to be fast, but when we have to choose between correctness and speed, we choose correctness.
Integration tests are run from an outsider's perspective. Outsiders see changes to the website -- nothing else. They cannot read any database, logfile, or other internal data. Integration tests can't, either.
Each test is one story, dealing with one feature. Each test is told from the user's perspective. It should be clear why the feature under test provides value to us -- money, fame, and so forth. If it's not clear how the feature gives us value, please comment the integration test to explain it.
First let's run them.
Set up the test suite:
- Install NodeJS.
npm
must be in your path. - Run
auto/setup-integration-tests.sh
from youroverview-server
checkout. This will run annpm install
command.
And now run the integration test suite:
./run # and keep it running in the background
auto/test-integration.sh
- The tests are written in CoffeeScript
- Test Framework: Mocha In under 50 words: Mocha defines
describe
,before
,after
,beforeEach
,afterEach
, andit
. They all share the same context: writing@variable
in abefore
block will let you read it in anit
block. - Asserters: Chai and Chai as Promised. This makes
.should
automatically defined on all objects. - Browser/Selenium Client: wd.js (see also: Selenium)
- Creating Promises: Q
- Helper functions: test/integration/lib and test/integration/support
- Integration Tests: test/integration/test
Each test looks like this:
describe 'Login', ->
...
it 'should log in', ->
@userBrowser
.tryLogIn(@userEmail, @userEmail)
.waitForElementByCss('.logged-in')
.shouldBeLoggedInAs(@userEmail)
.logOut()
Follow along in test/integration/test/LoginSpec.coffee.
Mocha is our integration test framework, which makes our tests look something like:
describe 'TestSuite name', ->
before ->
# Setup code that runs once before all tests
after ->
# Tear down code that runs once after all tests
beforeEach ->
# Setup code that runs before each test
# Since most of our setup loads web pages, which takes a long time,
# we tend to put our setup code in before
afterEach ->
# Teardown code. If you don't use beforeEach, you probably don't need to worry about this
it 'should test a specific feature', ->
# actual test code
It's also possible to nest tests, to take advantage of common setup
describe 'TestSuite name', ->
before -> # setup
after -> # teardown
it 'should test a specific feature', ->
# test code
describe 'the nested context', ->
before -> # only applies to this level
after -> # only applies to this level
it 'should also test this feature', ->
# test code that inherits the enclosing context defined by
# the top-level before and after
The before
and after
functions should ensure that the tests are (roughly) in the same state when they leave the tests as when they started. Practically, we try to ensure that the database is unchanged (newly created document sets are deleted, for example), but don't worry about the browser being at the same page.
See the Mocha documentation for details on how to do cool things like skip tests, or only run one test.
Mocha allows asynchronous tests. Here's how they normally work.
Synchronous:
it 'should test the + operator synchronously', ->
(1 + 1).should.equal(2)
Asynchronous:
it 'should test the + operator asynchronously', (done) -> # "done" is a callback
setTimeout(->
(1 + 1).should.equal(2)
done()
, 10) # delay the function for 10ms
When a Mocha test takes a callback as a parameter, the test won't complete until you call the callback. (No other tests will run, either.) This is essential for testing code that uses the event loop.
But we go one further, for simplicity. We use mocha-as-promised to extend Mocha and chai-as-promised to extend Chai, to give ourselves nicer syntax. With mocha-as-promised, any test that returns a promise will be asynchronous.
Q is one library that returns promises. (Anything with a .then
method can be a promise.) The Q folks have written a lovely guide on promises in JavaScript.
Less talk, more code? Okay. Here's some code using mocha-as-promised, chai-as-promised and Q:
it 'should test + with a promise', -> # notice there's no "done" callback
deferred = Q.deferred()
promise = deferred.promise # this has the ".then" method
setTimeout(->
deferred.resolve(1 + 1)
, 10) # delay 10ms
promise.should.eventually.equal(2) # return another promise
Notice that:
- The return value is a promise. So mocha-as-promised will ensure the test completes.
- The returned promise includes a Chai assertion.
eventually
indicates that the assertion operates on a promise and returns a promise.
Promises do two things:
- They hold a value
- They implicitly encode a time
You can "chain" promises together to ensure things happen in a certain order. For instance:
it 'should test + with a promise', -> # notice there's no "done" callback
deferred = Q.deferred()
setTimeout(->
deferred.resolve(1)
, 10) # delay 10ms
deferred.promise # first promise
.then((value) -> value + 1) # second promise
.then((value) -> value + 1) # third promise
.should.eventually.equal(3) # final promise
Don't you like how CoffeeScript indentation helps out with the dots? It's exactly as if the whitespace weren't there.
We like our tests to be readable, which sometimes requires the definition of nicely named helper functions. In order for the helper functions to work in the promise chains, they need to be added with testMethods.usingPromiseChainMethods
(support/testMethods)
testMethods = require('../support/testMethods')
describe 'Test a feature', ->
testMethods.usingPromiseChainMethods
doSomeComplexAction: ->
@ # refers to the browser that calls the method
.get(SomeUrl)
# do the steps needed for the complex action
it 'should test a feature', ->
@userBrowser
.doSomeComplexAction()
# rest of the test
If your test needs actions to be executed in two browsers, you need to combine the promises:
Sequentially:
@userBrowser
.doSomething()
.then(=> @adminBrowser.doSomethingAsAdmin())
Note that the return value of the then
is an @adminBrowser
promise, so any subsequent actions added in the chain will be run as @adminBrowser
, not @userBrowser
. To avoid confusion, it is best to not switch browsers except at the end of a promise chain.
In parallel: The Q library has many ways to create promises, including Q.all
Q = require('q')
Q.all([
@userBrowser.doSomething()
@adminBrowser.doSomethingAsAdmin()
])
The main help class is browser
(lib/browser.coffee
). A browser provides an interface to promise-generating methods that are standard to WD.js (and Selenium).
require('../lib/browser')
# Other test setup here
before -> @myBrowser = browser.create() # We start a Firefox browser
after -> @myBrowser.quit() # Kill the browser after tests
it 'should do browser stuff', ->
@myBrowser
.get(SomeUrl)
.waitForElementByCss('.some-button') # Use wait for non-instantaneous effects
.click()
.waitForElementByCss('.something-appearing-after-click').should.eventually.exist # Chai asserters
Rather than using waitForElementByCss
and its relatives, try our helper functions elementBy
and waitForElementBy
.
@myBrowser
.get(SomeUrl)
.waitForElementBy(tag: 'a', contains: 'click me', visible: true)
.click()
Both methods take the same arguments, and waitForElement
can be given an additional timeout parameter.
There arguments you can use are: name, value, class, className, contains, tag, visible (true by default). To specify a timeout use the form:
@myBrowser.waitForElementBy({class: 'button', name: 'History Reset'}, 10000) # specify timeout in ms
Look in the source for other functions that may be useful (like, listenForJqueryAjaxComplete
/waitForJqueryAjaxComplete
).
Here's a complete test file:
browser = require('../lib/browser')
describe 'google.com', ->
before -> # runs once for the entire file
@browser = browser.create() # a promise
after -> # runs once for the entire file
@browser.quit() # another promise
beforeEach -> # runs before each test
@browser # a promise!
.get("https://google.com") # a promise -- resolved on page load
it 'should have a search field', ->
@browser # a promise!
.elementByCss('[name=q]').should.eventually.exist # a promise
it 'should search without needing to click a button', ->
@browser # a promise!
.elementByCss('[name=q]').type('integration testing')
.waitForElementByXPath("//a[contains(em, 'Integration testing')]")
.should.eventually.be.fulfilled
There's lots to see here:
-
before()
,after()
andbeforeEach()
all operate on promises. - There are lots of new methods on the
@browser
promise:elementByCss()
,waitForElementByXPath()
,get()
, and so forth. Here's the wd.js API. - The
it
(test) code operates on the@browser
promise. That should raise warning bells: don't we need the promise returned from.get()
inbeforeEach()
? No. We don't need it, because the value of all the promises in the promise chain is identical: each value is a handle on the web browser. We only care about the timing; and we know that the promise inbeforeEach()
is resolved before theit
code begins, so the timing is fine. We could have stored a separate@browser1
promise inbeforeEach()
, but that would be excessive, since we know that@browser
and@browser1
would hold the exact same value when we reach theit
code. -
@browser.quit()
is important: it lets future tests re-use the browser. - We use PhantomJS as our default browser.
Back to our original example:
...
@userBrowser
.tryLogIn(@userEmail, @userEmail)
.waitForElementByCss('.logged-in')
.shouldBeLoggedInAs(@userEmail)
...
Where did .tryLogIn()
and .shouldBeLoggedInAs()
come from?
We defined them higher up in the file:
testMethods = require('../support/testMethods')
describe 'Login', ->
testMethods.usingPromiseChainMethods
tryLogIn: (email, password) ->
@waitForElementByCss('.session-form')
.elementByCss('.session-form [name=email]').type(email)
.elementByCss('.session-form [name=password]').type(password)
.elementByCss('.session-form [type=submit]').click()
shouldBeLoggedOut: () ->
@elementByCss('.session-form')
.should.eventually.exist
... # rest of the test file
testMethods.usingPromiseChainMethods
adds some methods to the WD.js promise chain. They are added in a before
call and removed in an after
call, so you cannot use them in subsequent after
blocks.
If your method is going to be useful in the entire unit test suite, consider adding it to lib/browser.coffee
.
Since most tests involve starting browsers and logging in as a user, asUser
(support/asUser.coffee) defines before and afters that take care of the common setup and tearDown. asUser provides
-
@userBrowser
- A browser logged in as a random user that is created (and deleted) for the test -
@adminBrowser
- A browser logged in as the default admin user.
By using asUser, you don't have to worry about creating or quitting the browser.
asUser = require('../support/asUser')
describe 'Test a feature', ->
asUser.usingTemporaryUser()
# no before/after needed to setup/teardown browser or temporary user
it 'should do something while logged in', ->
@userBrowser
.get(someUrl)
# rest of test
- If you haven't figured it out yet: tests are in
test/integration
. Runnpm test
from that directory to run them. - Want to run just one test? While you're editing a test, write
describe.only
instead ofdescribe
orit.only
instead ofit
. That will tell Mocha to ignore all other tests or blocks. (Remember to remove the.only
before committing!) - Better: run
npm install -g mocha
and then you can run tests with a plain oldmocha
. That's useful because you can add--grep
: for instance,mocha --grep Login
. - Your
it
code should be nothing but promise chains. Use helper methods for anything confusing. - To quickly test XPath strings, such as you'd pass to
.elementByXPath(...)
, use Chrome's built-in$x(...)
in the JavaScript Console, which works identically. - To quickly test CSS selectors, such as you'd pass to
.elementByCss(...)
, use Chrome's built-in$$(...)
in the JavaScript Console, which works identically. - Beware promises'
.then
method: it can be handy, but the promises it returns don't have any WebDriver methods.
For instance, this won't work:
it 'should work', ->
@browser
.get("/some/url")
.then(-> console.log("Got the url"))
.elementByCss("[name=name]").should.eventually.exist # Promise doesn't have .elementByCss method
.then
returns a promise (so if it's at the end of the chain, mocha-as-promised will run as expected). But that promise doesn't have any WebDriver methods: only WebDriver methods return promises that are enhanced with WebDriver methods.
But this will work:
it 'should work', ->
@browser1
.get("/some/url")
.then(=> @browser2.get("/some/other/url"))
.elementByCss("[name=name]").should.eventually.exist
That's because the promise returned in the .then
call is actually a WebDriver promise. It's a promise returned from @browser2
, so .elementByCss
will be called on @browser2
.
And for debug logging in particular, consider .print(...)
instead of .then(-> console.log(...))
. .print(...)
will output ...
followed by the promise's result.
Our tests must uphold these requirements. This is a reasonable balance between simplicity and speed.
- Start browsers in
before
blocks and quit them inafter
blocks (that is, reuse the same browser for all tests in a file). - In general, create objects in
before
blocks and delete them inafter
blocks. For instance, if you're testing a DocumentSet, create it once and delete it once. If you're logging in as a user, create the user once and delete it once. - You can nest
describe
blocks. That works very well withbefore
andafter
. - Each block must leave things as it found them. For instance, if you're testing login and each test is meant to start logged out, then make sure each test ends logged out.
- All browsers must be cleaned, with
.deleteAllCookies().quit()
. (deleteAllCookies()
requires a page to be loaded; by putting it at the end of all test suites instead of the beginning, we save a page load per suite.)
It would be cleaner to use beforeEach
and afterEach
exclusively. Unfortunately, page loads take hundreds of milliseconds (even seconds, in dev environments, as RequireJS fetches every JavaScript file individually). We should try and avoid unnecessary page requests, when that doesn't make the code too messy.