Merge pull request #22 from ArnaudBuchholz/1.1.0

1.1.0

README.md

@@ -9,15 +9,15 @@
 [![MIT License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_shield)
 
-A test runner for UI5 applications enabling parallel execution of tests.
+A self-sufficient test runner for UI5 applications enabling parallel execution of tests.
 
-> To put it in a nutshell, some applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it crashes. JavaScript is using garbage collecting but it needs time to operate and the stress caused by executing the tests does not let enough bandwidth for the browser to free up the memory.
+> To put it in a nutshell, some applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it blows up. JavaScript is using garbage collecting but it needs time to operate and the stress caused by executing the tests does not let enough bandwidth for the browser to free up the memory.
 
 > This tool is designed and built as a **substitute** of the [UI5 karma runner](https://github.com/SAP/karma-ui5). It executes all the tests in **parallel** thanks to several browser instances *(which also **reduces the total execution time**)*.
 
 ## Documentation
 
-* Concept is detailed in the  article [REserve - Testing UI5](https://arnaud-buchholz.medium.com/reserve-testing-ui5-85187d5eb7f1)
+* Initial concept is detailed in the  article [REserve - Testing UI5](https://arnaud-buchholz.medium.com/reserve-testing-ui5-85187d5eb7f1)
 * Tool was presented during UI5Con'21 : [A different approach to UI5 tests execution](https://youtu.be/EBp0bdIqu4s)
 
 ## How to install
@@ -29,8 +29,11 @@ A test runner for UI5 applications enabling parallel execution of tests.
 
 * Clone the project [training-ui5con18-opa](https://github.com/ArnaudBuchholz/training-ui5con18-opa) and run `npm install`
 * Inside the project, use `npm run karma` to test with the karma runner
-* Inside the project, use `ui5-test-runner -port:8080 -ui5:https://ui5.sap.com/1.87.0/ -cache:.ui5`
-* You may follow the progress of the test executions using http://localhost:8080/_/progress.html
+* Inside the project, use `ui5-test-runner -port:8080 -ui5:https://ui5.sap.com/1.87.0/ -cache:.ui5 -keepAlive`
+* Follow the progress of the test executions using http://localhost:8080/_/progress.html
+* When the tests are completed, inspect the results by opening :
+  - http://localhost:8080/_/report.html
+  - http://localhost:8080/_/coverage/lcov-report/index.html
 
 ## How to use
 
@@ -69,7 +72,7 @@ The list of options is detailed below but to explain the command :
 
 * `-cache:.ui5` : caches UI5 resources to boost loading of pages. It stores resources in a project folder named `.ui5` *(you may use an absolute path if preferred)*.
 
-* `-libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/` : maps the library path (access to URL `/resources/my/namespace/feature/lib/library.js` will be mapped to the file path `./my.namespace.feature.project.lib/src/my/namespace/feature/lib/library.js`)
+* `-libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/` : maps the library path (access to URL `/resources/my/namespace/feature/lib/library.js` will be mapped to the file path `../my.namespace.feature.project.lib/src/my/namespace/feature/lib/library.js`)
 
 You may also use :
 * `-ui5:https://ui5.sap.com/1.92.1/` : uses a specific version of UI5
@@ -78,7 +81,7 @@ You may also use :
 
 * `"-args:__URL__ __REPORT__ --visible"` : changes the browser spawning command line to make the browser windows **visible**
 
-* `-parallel:3` : increases *(changes)* the number of parallel execution *(by default it uses 2)*. You may even use `0` to only serve the application *(but the tests are not executed)*.
+* `-parallel:3` : increases *(changes)* the number of parallel execution *(by default it uses 2)*. You may even use `0` to only serve the application *(the tests are not executed)*.
 
 * `-keepAlive` : the server remains active after executing the tests
 
@@ -137,6 +140,7 @@ You may also use :
 | browser | *String, see description* | Browser instantiation command, it should point to a node.js script *(absolute or relative to `cwd`)*.<br/>By default, a script will instantiate chromium through puppetteer |
 | browserRetry | `1` | Browser instantiation retries : if the command **fails** unexpectedly, it is re-executed *(`0` means no retry)*.<br/>The page **fails** if **all attempts** fail |
 | args | `'__URL__ __REPORT__'` | Browser instantiation arguments :<ul><li>`'__URL__'` is replaced with the URL to open</li><li>`'__REPORT__'` is replaced with a folder path that is associated with the current URL *(can be used to store additional traces such as console logs or screenshots)*</li><li>`'__RETRY__'` is replaced with the retry count *(0 for the first execution, can be used to put additional traces or change strategy)*</i>*</li></ul> |
+| noScreenshot | `false` | No screenshot is taken during the tests execution (faster if the browser command supports screenshot) |
 | -- | | Parameters given right after `--` are directly added to the browser instantiation arguments *(see below)* |
 | parallel | `2` | Number of parallel tests executions (`0` to ignore tests and keep alive) |
 | tstReportDir | `'report'` | Directory to output test reports *(relative to `cwd`)* |
@@ -187,10 +191,24 @@ For instance :
 
 > Structure of the `libs` parameter
 
+## Tips & tricks
+
+* The runner takes a screenshot for **every** OPA assertion (`Opa5.assert.ok`)
+* To benefit from **parallelization**, split the OPA test pages per journey.<br> An example pattern :
+  - **Declare** the list of journeys in a json file : [`AllJourneys.json`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.json)
+  - **Enumerate** `AllJourneys.json` in [`testsuite.qunit.html`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/testsuite.qunit.html#L17) to declare as many pages as journeys
+  - **Modify** [`AllJourneys.js`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.js#L26) to support a `journey=` parameter and list all declared journeys
+
 ## Building a custom browser instantiation command
 
 * You may follow the pattern being used by [`chromium.js`](https://github.com/ArnaudBuchholz/ui5-test-runner/blob/main/defaults/chromium.js)
 * It is **mandatory** to ensure that the child process explicitly exits at some point *(see this [thread](https://github.com/nodejs/node-v0.x-archive/issues/2605) explaining the fork behavior with Node.js)*
+* The child process will receive messages that must be handled appropriately :
+  - `message.command === 'stop'` : the browser must be closed and the command line must exit
+  - `message.command === 'capabilities'` : a message must be sent back to indicate if the following features are supported *(boolean)*
+    - `screenshot` : the browser can take screenshots (in the `__REPORT__` folder, name is provided when needed)
+    - `consoleLog` : the browser serializes the console traces (in the `__REPORT__` folder with the name `console.txt`)
+  - `message.command === 'screenshot'` : should generate a screenshot (the message contains a `filename` member). To indicate that the screenshot is done, the command line must send back the same message (`process.send(message)`).
 
 ## License
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_large)

__mocks__/child_process.js

@@ -1,9 +1,12 @@
 const EventEmitter = require('events')
 const _hook = new EventEmitter()
 
+class Channel extends EventEmitter {
+}
+
 class ChildProcess extends EventEmitter {
   send (message) {
-    this.emit('message', message)
+    this.emit('message.received', message)
   }
 
   close () {
@@ -15,14 +18,18 @@ class ChildProcess extends EventEmitter {
   get args () { return this._args }
   get options () { return this._options }
   get connected () { return this._connected }
+  get stdout () { return this._stdout }
+  get stderr () { return this._stderr }
 
   constructor (scriptPath, args, options) {
     super()
     this._connected = true
     this._scriptPath = scriptPath
     this._args = args
     this._options = options
-    setTimeout(() => _hook.emit('new', this), 0) // Defer the call since creation is 'asynchronous'
+    this._stdout = new Channel()
+    this._stderr = new Channel()
+    _hook.emit('new', this)
   }
 }
 

__tests__/src/browser.js

@@ -1,37 +1,55 @@
 jest.mock('child_process')
+jest.mock('../../src/output', () => {
+  const EventEmitter = require('events')
+  class Output extends EventEmitter {
+    browserStart (...args) { this.emit('browserStart', ...args) }
+    browserStopped (...args) { this.emit('browserStopped', ...args) }
+    browserCapabilities (...args) { this.emit('browserCapabilities', ...args) }
+    browserTimeout (...args) { this.emit('browserTimeout', ...args) }
+    browserRetry (...args) { this.emit('browserRetry', ...args) }
+    browserClosed (...args) { this.emit('browserClosed', ...args) }
+    monitor (...args) { this.emit('monitor', ...args) }
+  }
+  return new Output()
+})
+const output = require('../../src/output')
 const { join } = require('path')
 const { _hook: hook } = require('child_process')
 const jobFactory = require('../../src/job')
-const { start, stop } = require('../../src/browsers')
+const { start, stop, screenshot } = require('../../src/browsers')
 
 const cwd = '/test/project'
 
-describe('src/job', () => {
+describe('src/browser', () => {
   let log
+  let warn
+  let error
   let job
 
   beforeAll(() => {
     log = jest.spyOn(console, 'log').mockImplementation()
+    warn = jest.spyOn(console, 'warn').mockImplementation()
+    error = jest.spyOn(console, 'error').mockImplementation()
     job = jobFactory.fromCmdLine(cwd, [0, 0, `-tstReportDir:${join(__dirname, '../../tmp/browser')}`])
   })
 
   it('starts returns a promise resolved on stop', () => {
     hook.once('new', childProcess => {
-      stop(job, 'test.html')
+      setTimeout(() => stop(job, 'test.html'), 0)
     })
     return start(job, 'test.html')
   })
 
   it('stops automatically after a timeout', () => {
-    let done
+    let stopReceived
     const waitingForMessage = new Promise(resolve => {
-      done = resolve
+      stopReceived = resolve
     })
     hook.once('new', childProcess => {
-      childProcess.on('message', message => {
+      childProcess.on('message.received', message => {
         if (message.command === 'stop') {
           childProcess.close()
-          done()
+          stopReceived()
         }
       })
     })
@@ -42,6 +60,78 @@ describe('src/job', () => {
     ])
   })
 
+  it('queries capabilities', () => {
+    output.once('browserCapabilities', capabilities => {
+      expect(capabilities.uid).toStrictEqual(123)
+      stop(job, 'test.html')
+    })
+    hook.once('new', childProcess => {
+      childProcess.on('message.received', message => {
+        if (message.command === 'capabilities') {
+          childProcess.emit('message', {
+            command: 'capabilities',
+            screenshot: true,
+            consoleLog: true,
+            uid: 123
+          })
+        }
+      })
+    })
+    return start(job, 'test.html')
+  })
+
+  it('supports screenshot', () => {
+    hook.once('new', childProcess => {
+      childProcess.on('message.received', message => {
+        if (message.command === 'screenshot') {
+          expect(message.filename).toStrictEqual('screenshot.png')
+          setTimeout(() => {
+            childProcess.emit('message', message)
+          }, 0)
+        }
+      })
+      setTimeout(async () => {
+        await screenshot(job, 'test.html', 'screenshot.png')
+        stop(job, 'test.html')
+      }, 0)
+    })
+    return start(job, 'test.html')
+  })
+
+  it('supports screenshot (noScreenshot)', () => {
+    job.noScreenshot = true
+    hook.once('new', childProcess => {
+      childProcess.on('message.received', message => {
+        expect(message.command).not.toStrictEqual('screenshot')
+      })
+      setTimeout(async () => {
+        await screenshot(job, 'test.html', 'screenshot.png')
+        job.noScreenshot = false
+        stop(job, 'test.html')
+      }, 0)
+    })
+    return start(job, 'test.html')
+  })
+
+  it('supports screenshot (page does not exist)', async () => {
+    await expect(screenshot(job, 'test2.html', 'screenshot.png')).resolve
+  })
+
+  it('supports screenshot (page disconnected)', async () => {
+    job.browserCapabilities = { screenshot: true }
+    hook.once('new', childProcess => {
+      childProcess.on('message.received', message => {
+        expect(message.command).not.toStrictEqual('screenshot')
+      })
+      setTimeout(async () => {
+        childProcess._connected = false
+        await screenshot(job, 'test.html', 'screenshot.png')
+        stop(job, 'test.html')
+      }, 0)
+    })
+    return start(job, 'test.html')
+  })
+
   it('retries automatically if the process crashes unexpectedly (second succeeds)', () => {
     let step = 0
     hook.once('new', childProcess => {
@@ -81,6 +171,11 @@ describe('src/job', () => {
   })
 
   afterAll(() => {
+    expect(log.mock.calls.length).toStrictEqual(0)
+    expect(warn.mock.calls.length).toStrictEqual(0)
+    expect(error.mock.calls.length).toStrictEqual(0)
     log.mockRestore()
+    warn.mockRestore()
+    error.mockRestore()
   })
 })

__tests__/src/coverage.js

@@ -1,4 +1,10 @@
 jest.mock('child_process')
+jest.mock('../../src/output', () => {
+  return {
+    nyc: jest.fn(),
+    monitor: jest.fn()
+  }
+})
 const { join } = require('path')
 const jobFactory = require('../../src/job')
 const { _hook: hook } = require('child_process')
@@ -10,10 +16,14 @@ const cwd = '/test/project'
 
 describe('src/coverage', () => {
   let log
+  let warn
+  let error
   const coveragePath = join(__dirname, '../../tmp/coverage')
 
   beforeAll(() => {
     log = jest.spyOn(console, 'log').mockImplementation()
+    warn = jest.spyOn(console, 'warn').mockImplementation()
+    error = jest.spyOn(console, 'error').mockImplementation()
     return cleanDir(coveragePath)
   })
 
@@ -71,7 +81,7 @@ describe('src/coverage', () => {
       let triggered = false
       hook.once('new', childProcess => {
         triggered = true
-        childProcess.emit('close')
+        setTimeout(() => childProcess.emit('close'), 0)
       })
       await instrument(job)
       expect(triggered).toStrictEqual(true)
@@ -88,7 +98,7 @@ describe('src/coverage', () => {
         } else if (childProcess.args[0] === 'report') {
           reportTriggered = true
         }
-        childProcess.emit('close')
+        setTimeout(() => childProcess.emit('close'), 0)
       }
       hook.on('new', newChildProcess)
       await generateCoverageReport(job)
@@ -104,6 +114,11 @@ describe('src/coverage', () => {
   })
 
   afterAll(() => {
+    expect(log.mock.calls.length).toStrictEqual(0)
+    expect(warn.mock.calls.length).toStrictEqual(0)
+    expect(error.mock.calls.length).toStrictEqual(0)
     log.mockRestore()
+    warn.mockRestore()
+    error.mockRestore()
   })
 })

__tests__/src/job.js

@@ -1,13 +1,21 @@
+jest.mock('../../src/output', () => {
+  return {
+    unexpectedOptionValue: jest.fn()
+  }
+})
 const { dirname, join } = require('path')
-
 const jobFactory = require('../../src/job')
 const cwd = '/test/project'
 const normalizePath = path => path.replace(/\\/g, '/') // win -> unix
 
 describe('src/job', () => {
+  let log
+  let warn
   let error
 
   beforeAll(() => {
+    log = jest.spyOn(console, 'log').mockImplementation()
+    warn = jest.spyOn(console, 'warn').mockImplementation()
     error = jest.spyOn(console, 'error').mockImplementation()
   })
 
@@ -86,11 +94,8 @@ describe('src/job', () => {
   })
 
   it('fixes invalid browserRetry value', () => {
-    const warn = jest.spyOn(console, 'warn').mockImplementation()
     const job = jobFactory.fromCmdLine(cwd, [0, 0, '-browserRetry: -1'])
     expect(job.browserRetry).toStrictEqual(1)
-    expect(warn.mock.calls.length).toStrictEqual(1)
-    warn.mockRestore()
   })
 
   it('preloads settings from ui5-test-runner.json', () => {
@@ -120,6 +125,11 @@ describe('src/job', () => {
   })
 
   afterAll(() => {
+    expect(log.mock.calls.length).toStrictEqual(0)
+    expect(warn.mock.calls.length).toStrictEqual(0)
+    expect(error.mock.calls.length).toStrictEqual(0)
+    log.mockRestore()
+    warn.mockRestore()
     error.mockRestore()
   })
 })