Skip to content

Writing integration tests

Jump to bottom
Jonathan Stray edited this page Apr 4, 2017 · 5 revisions

Yay, you're going to help us test our entire software stack to make sure all components work together? We want this to be fast, but when we have to choose between correctness and speed, we choose correctness.

Integration tests are run from an outsider's perspective. Outsiders see changes to the website -- nothing else. They cannot read any database, logfile, or other internal data. Integration tests can't, either.

Each test is one story, dealing with one feature. Each test is told from the user's perspective. It should be clear why the feature under test provides value to us -- money, fame, and so forth. If it's not clear how the feature gives us value, please comment the integration test to explain it.

First let's run them.

Running Integration tests

Set up the test suite:

  1. Install NodeJS. npm must be in your path.
  2. Run auto/setup-integration-tests.sh from your overview-server checkout. This will run an npm install command.

And now run the integration test suite:

./run # and keep it running in the background
auto/test-integration.sh

The integration test stack

What a test looks like

Each test looks like this:

describe 'Login', ->
  ...
  it 'should log in', ->
    @userBrowser
      .tryLogIn(@userEmail, @userEmail)
      .waitForElementByCss('.logged-in')
      .shouldBeLoggedInAs(@userEmail)
      .logOut()

Follow along in test/integration/test/LoginSpec.coffee.

Mocha

Mocha is our integration test framework, which makes our tests look something like:

describe 'TestSuite name', ->
  before -> 
    # Setup code that runs once before all tests

  after -> 
    # Tear down code that runs once after all tests

  beforeEach -> 
     # Setup code that runs before each test
     # Since most of our setup loads web pages, which takes a long time, 
     # we tend to put our setup code in before

  afterEach ->
     # Teardown code. If you don't use beforeEach, you probably don't need to worry about this

  it 'should test a specific feature', ->
     # actual test code

It's also possible to nest tests, to take advantage of common setup

describe 'TestSuite name', ->
  before ->  # setup
  after ->   # teardown

  it 'should test a specific feature', ->
    # test code

  describe 'the nested context', ->
    before -> # only applies to this level
    after -> # only applies to this level

    it 'should also test this feature', ->
      # test code that inherits the enclosing context defined by 
      # the top-level before and after

The before and after functions should ensure that the tests are (roughly) in the same state when they leave the tests as when they started. Practically, we try to ensure that the database is unchanged (newly created document sets are deleted, for example), but don't worry about the browser being at the same page. See the Mocha documentation for details on how to do cool things like skip tests, or only run one test.

Asynchronicity

Mocha allows asynchronous tests. Here's how they normally work.

Synchronous:

  it 'should test the + operator synchronously', ->
    (1 + 1).should.equal(2)

Asynchronous:

  it 'should test the + operator asynchronously', (done) -> # "done" is a callback
    setTimeout(->
      (1 + 1).should.equal(2)
      done()
    , 10) # delay the function for 10ms

When a Mocha test takes a callback as a parameter, the test won't complete until you call the callback. (No other tests will run, either.) This is essential for testing code that uses the event loop.

But we go one further, for simplicity. We use mocha-as-promised to extend Mocha and chai-as-promised to extend Chai, to give ourselves nicer syntax. With mocha-as-promised, any test that returns a promise will be asynchronous.

Q is one library that returns promises. (Anything with a .then method can be a promise.) The Q folks have written a lovely guide on promises in JavaScript.

Less talk, more code? Okay. Here's some code using mocha-as-promised, chai-as-promised and Q:

  it 'should test + with a promise', -> # notice there's no "done" callback
    deferred = Q.deferred()
    promise = deferred.promise # this has the ".then" method

    setTimeout(->
      deferred.resolve(1 + 1)
    , 10) # delay 10ms

    promise.should.eventually.equal(2) # return another promise

Notice that:

  • The return value is a promise. So mocha-as-promised will ensure the test completes.
  • The returned promise includes a Chai assertion. eventually indicates that the assertion operates on a promise and returns a promise.

Promise chains

Promises do two things:

  1. They hold a value
  2. They implicitly encode a time

You can "chain" promises together to ensure things happen in a certain order. For instance:

  it 'should test + with a promise', -> # notice there's no "done" callback
    deferred = Q.deferred()

    setTimeout(->
      deferred.resolve(1)
    , 10) # delay 10ms

    deferred.promise              # first promise
      .then((value) -> value + 1) # second promise
      .then((value) -> value + 1) # third promise
      .should.eventually.equal(3) # final promise

Don't you like how CoffeeScript indentation helps out with the dots? It's exactly as if the whitespace weren't there.

MYO helper functions

We like our tests to be readable, which sometimes requires the definition of nicely named helper functions. In order for the helper functions to work in the promise chains, they need to be added with testMethods.usingPromiseChainMethods (support/testMethods)

testMethods = require('../support/testMethods')

describe 'Test a feature', ->
  testMethods.usingPromiseChainMethods
    doSomeComplexAction: ->
      @    # refers to the browser that calls the method
        .get(SomeUrl)
        # do the steps needed for the complex action

  it 'should test a feature', ->
    @userBrowser
      .doSomeComplexAction()
      # rest of the test

Promise tricks

If your test needs actions to be executed in two browsers, you need to combine the promises:

Sequentially:

@userBrowser
    .doSomething()
    .then(=> @adminBrowser.doSomethingAsAdmin())

Note that the return value of the then is an @adminBrowser promise, so any subsequent actions added in the chain will be run as @adminBrowser, not @userBrowser. To avoid confusion, it is best to not switch browsers except at the end of a promise chain.

In parallel: The Q library has many ways to create promises, including Q.all

Q = require('q')

Q.all([
  @userBrowser.doSomething()
  @adminBrowser.doSomethingAsAdmin()
])

WebDriver

The main help class is browser (lib/browser.coffee). A browser provides an interface to promise-generating methods that are standard to WD.js (and Selenium).

require('../lib/browser')

# Other test setup here
before -> @myBrowser = browser.create()  # We start a Firefox browser
after -> @myBrowser.quit() # Kill the browser after tests

it 'should do browser stuff', ->
  @myBrowser
    .get(SomeUrl)
    .waitForElementByCss('.some-button') # Use wait for non-instantaneous effects
    .click()
    .waitForElementByCss('.something-appearing-after-click').should.eventually.exist # Chai asserters

Rather than using waitForElementByCss and its relatives, try our helper functions elementBy and waitForElementBy.

@myBrowser
  .get(SomeUrl)
  .waitForElementBy(tag: 'a', contains: 'click me', visible: true)
  .click()

Both methods take the same arguments, and waitForElement can be given an additional timeout parameter. There arguments you can use are: name, value, class, className, contains, tag, visible (true by default). To specify a timeout use the form:

 @myBrowser.waitForElementBy({class: 'button', name: 'History Reset'}, 10000) # specify timeout in ms

Look in the source for other functions that may be useful (like, listenForJqueryAjaxComplete/waitForJqueryAjaxComplete).

Here's a complete test file:

browser = require('../lib/browser')

describe 'google.com', ->
  before -> # runs once for the entire file
    @browser = browser.create() # a promise

  after -> # runs once for the entire file
    @browser.quit() # another promise

  beforeEach -> # runs before each test
    @browser # a promise!
      .get("https://google.com") # a promise -- resolved on page load

  it 'should have a search field', ->
    @browser # a promise!
      .elementByCss('[name=q]').should.eventually.exist # a promise

  it 'should search without needing to click a button', ->
    @browser # a promise!
      .elementByCss('[name=q]').type('integration testing')
      .waitForElementByXPath("//a[contains(em, 'Integration testing')]")
      .should.eventually.be.fulfilled

There's lots to see here:

  • before(), after() and beforeEach() all operate on promises.
  • There are lots of new methods on the @browser promise: elementByCss(), waitForElementByXPath(), get(), and so forth. Here's the wd.js API.
  • The it (test) code operates on the @browser promise. That should raise warning bells: don't we need the promise returned from .get() in beforeEach()? No. We don't need it, because the value of all the promises in the promise chain is identical: each value is a handle on the web browser. We only care about the timing; and we know that the promise in beforeEach() is resolved before the it code begins, so the timing is fine. We could have stored a separate @browser1 promise in beforeEach(), but that would be excessive, since we know that @browser and @browser1 would hold the exact same value when we reach the it code.
  • @browser.quit() is important: it lets future tests re-use the browser.
  • We use PhantomJS as our default browser.

Custom methods

Back to our original example:

    ...
    @userBrowser
      .tryLogIn(@userEmail, @userEmail)
      .waitForElementByCss('.logged-in')
      .shouldBeLoggedInAs(@userEmail)
    ...

Where did .tryLogIn() and .shouldBeLoggedInAs() come from?

We defined them higher up in the file:

testMethods = require('../support/testMethods')

describe 'Login', ->
  testMethods.usingPromiseChainMethods
    tryLogIn: (email, password) ->
      @waitForElementByCss('.session-form')
        .elementByCss('.session-form [name=email]').type(email)
        .elementByCss('.session-form [name=password]').type(password)
        .elementByCss('.session-form [type=submit]').click()

    shouldBeLoggedOut: () ->
      @elementByCss('.session-form')
        .should.eventually.exist

  ... # rest of the test file

testMethods.usingPromiseChainMethods adds some methods to the WD.js promise chain. They are added in a before call and removed in an after call, so you cannot use them in subsequent after blocks.

If your method is going to be useful in the entire unit test suite, consider adding it to lib/browser.coffee.

asUser

Since most tests involve starting browsers and logging in as a user, asUser (support/asUser.coffee) defines before and afters that take care of the common setup and tearDown. asUser provides

  • @userBrowser - A browser logged in as a random user that is created (and deleted) for the test
  • @adminBrowser - A browser logged in as the default admin user.

By using asUser, you don't have to worry about creating or quitting the browser.

asUser = require('../support/asUser')

describe 'Test a feature', ->

  asUser.usingTemporaryUser()

  # no before/after needed to setup/teardown browser or temporary user

  it 'should do something while logged in', ->
    @userBrowser
      .get(someUrl)
      # rest of test

Tips for editing tests

  • If you haven't figured it out yet: tests are in test/integration. Run npm test from that directory to run them.
  • Want to run just one test? While you're editing a test, write describe.only instead of describe or it.only instead of it. That will tell Mocha to ignore all other tests or blocks. (Remember to remove the .only before committing!)
  • Better: run npm install -g mocha and then you can run tests with a plain old mocha. That's useful because you can add --grep: for instance, mocha --grep Login.
  • Your it code should be nothing but promise chains. Use helper methods for anything confusing.
  • To quickly test XPath strings, such as you'd pass to .elementByXPath(...), use Chrome's built-in $x(...) in the JavaScript Console, which works identically.
  • To quickly test CSS selectors, such as you'd pass to .elementByCss(...), use Chrome's built-in $$(...) in the JavaScript Console, which works identically.
  • Beware promises' .then method: it can be handy, but the promises it returns don't have any WebDriver methods.

For instance, this won't work:

  it 'should work', ->
    @browser
      .get("/some/url")
      .then(-> console.log("Got the url"))
      .elementByCss("[name=name]").should.eventually.exist # Promise doesn't have .elementByCss method

.then returns a promise (so if it's at the end of the chain, mocha-as-promised will run as expected). But that promise doesn't have any WebDriver methods: only WebDriver methods return promises that are enhanced with WebDriver methods.

But this will work:

  it 'should work', ->
    @browser1
      .get("/some/url")
      .then(=> @browser2.get("/some/other/url"))
      .elementByCss("[name=name]").should.eventually.exist

That's because the promise returned in the .then call is actually a WebDriver promise. It's a promise returned from @browser2, so .elementByCss will be called on @browser2.

And for debug logging in particular, consider .print(...) instead of .then(-> console.log(...)). .print(...) will output ... followed by the promise's result.

The test contract in Overview

Our tests must uphold these requirements. This is a reasonable balance between simplicity and speed.

  • Start browsers in before blocks and quit them in after blocks (that is, reuse the same browser for all tests in a file).
  • In general, create objects in before blocks and delete them in after blocks. For instance, if you're testing a DocumentSet, create it once and delete it once. If you're logging in as a user, create the user once and delete it once.
  • You can nest describe blocks. That works very well with before and after.
  • Each block must leave things as it found them. For instance, if you're testing login and each test is meant to start logged out, then make sure each test ends logged out.
  • All browsers must be cleaned, with .deleteAllCookies().quit(). (deleteAllCookies() requires a page to be loaded; by putting it at the end of all test suites instead of the beginning, we save a page load per suite.)

It would be cleaner to use beforeEach and afterEach exclusively. Unfortunately, page loads take hundreds of milliseconds (even seconds, in dev environments, as RequireJS fetches every JavaScript file individually). We should try and avoid unnecessary page requests, when that doesn't make the code too messy.

Clone this wiki locally