Skip to content

Latest commit

 

History

History
476 lines (361 loc) · 46.8 KB

DEVELOPER.md

File metadata and controls

476 lines (361 loc) · 46.8 KB

Eyra developer's guide

This document describes the software in more detail, steps to take when using the software/making modifications, a description of how to use the quality control, and how the different components are linked together.

A recommended read as well is the article published on this software, which can be found at Docs/Petursson_et_al_2016.pdf. It describes many design criteria along with how the software operates and more.

Development

The Stack

  • MySQL
  • Python
    • Flask
  • JavaScript
  • Celery
  • Kaldi
    • Marosijo
  • bash
  • Android

Which platforms has Eyra been compiled on?

Debian 8 Jessie

Ubuntu Server 14.04, 16.04

Android 6.0 and 5.1.1

also works on Firefox and Chrome

Short description of folder structure

  • AndroidApp
    The entire Android app, java code and all. IDE used is Android Studio.

  • Backend
    The Flask python code, which handles connections to the MySQL database among other things. Also includes the schema for the database, the entire quality control (QC) and sql code needed for setup. Recordings are saved in /data/eyra/recordings/ by default (this can be changed in app.py by changing app.config['MAIN_RECORDINGS_PATH'], should be an absolute path). Number of useful scripts in scripts/ and qc/scripts/.

  • Docs
    Some additional documentation to this guide and the README.md.

  • Frontend
    The AngularJS code and all related. deploy application using grunt deploy in the da-webapp/ folder (or ./Setup/setup.sh --frontend-app from project root). Work in src/ is then compiled into app/.

  • Local
    Locally generated code, generated by the Setup/setup.sh script. Used mainly for the apache server.

  • Setup
    Setup of the app. Running setup.sh installs the webapp from scratch. Includes code for all the components needing a setup, e.g. the apache server, the database setup, the frontend setup and the QC.
    Note: Inserting of the prompts is commented out in erase_and_rewind.sql since it is assumed you will be using your own prompts.

Detailed description of the components

This list is not exhaustive.

  • Android app
    The Android app was created with the sole purpose of being a workaround for the observed glitch where recordings through a browser on a phone gave us recordings with a sample rate of 48kHz, but appeared to only contain data up to 16kHz.
    It does have some pros like easing the acquisition of a device id, but for the most part an incredibly unfortunate resort so far (makes the entire user process much more complicated).

    The setup is a typical auto-setup of a simple app. Here is a list of some of the components, by no means comprehensive

  • Backend
    The entire backend except for the apache server.

    • db/
      Running ./Setup/setup.sh --mysqldb typically sets up the entire database.
      • invalidate_tokens.sql
        This is for the optional feature of marking certain tokens (prompts) invalid in our database. This is because changing the prompt database is hard work (some devices might still have old prompts, and so the ids would all get messed up if you simply deleted the old one and added a new one). So, you can set the valid property as FALSE in the table token for each token you want to stop serving to your devices, but you still maintain its id in the database for references.
        Look at scripts/{generate_invalid_token_ids.sh,generate_invalidate_tokens.sh} for how to make this file. You have to uncomment the source invalidate_tokens.sql line in erase_and_rewind.sql if you want to utilize this during setup.
      • permissions.sql
        During development, you typically access the database locally as root (mysql -u root -p), but the code uses a user called default without a password which has to have the correct permissions (and no more, just a security concern).
      • schema_setup.sql
        The main file here is schema_setup.sql which as you might suspect contains the layout of our database.
    • scripts/
      Number of useful scripts here, should be for the most part self-documenting. Including but not limited to
      • tokens_to_sql.py
        If you have a list of prompts (tokens) in a file ready to make decoding graphs, i.e. a key, space and then the prompt as per malromur_toks.txt (an example), then you can convert them to the required malromur_toks.sql file running this command: 
        python3 tokens_to_sql.py <(cut -f 2- -d' ' ../lang_data/is/malromur_toks.txt) ../db/malromur_toks.sql
      • convert_to_eyra_database/
        Which contains code to migrate from other databases to Eyra, you can use it for reference if you need to do the same. Note though, that these migrations might only apply to an earlier version of Eyra, depending on when they were done (these are obviously not maintained).
      • firebase/
        Code to recover data uploaded as backup to firebase.
    • server-interface/
      Set up as a pretty basic Flask application. Also contains the entire QC (see large section below the Development section). Entry code is app.py which handles the routes to implement the ClientServerAPI.md and all related. An attempt is made to separate the logic using classes in the *_handler.py's. The db_handler uses code from config.py to connect to our MySQL database.
      • auth_handler.py
        Contains allowed users to login as admins (under Settings in the client app).
      • send_mail.py
        This is a potentially useful script used to send out an email summary of all gathered data with Eyra on the computer where the mail is sent from. Has all sorts of statistically relevant info, total recordings gathered, recordings by speaker, gender, device etc. Could for example put in a cronjob to send it out every X hours to interested parties for your data collection.
    • lang_data/
      Here you can place any language specific data like a lexicon for example.
      • is/
        Example data in Icelandic.

  • Frontend
    An AngularJS application and the main component (what the user sees ;)) of Eyra. Setup used the angular-seed code to start with. Some original design philosophies:

    • Favor simplicity over performance unless otherwise needed.
    • Try to keep logic in services and not in controllers.
    • A single controller for each view.
    • App had to be able to be completely offline (appcache was used, should be upgraded to Service Workers, see issue #60).
    • RESTful communication with server.

    Overall documentation is lacking issue #41.

    • da-webapp/
      A completely useless distinction to have this da-webapp folder :( until they add more components to the Frontend! Stands for "data acquisition webapp". Data acquisition was the project's original title, until Simon came up with Eyra.
      • src/
        The source, code from here is compiled into the app/ folder on a ./Setup/setup.sh --frontend-app or a grunt deploy run. However, with recent sass updates, which grunt depends on, now Eyra only works with NodeJS >=v4. NodeJS v0.10.x or v0.12.x will error out.. So make sure you have the correct version of NodeJS. Under normal conditions, the application is run from the app/ folder. Should be pretty straightforward where stuff is here. See Run webapp straight from Frontend/da-webapp/src in the Some useful info section below on how to work straight from source instead of having to compile into app/ first. Uses Mozilla's localForage to store data in the browsers databases on the client side (for offline use, e.g. store cached recordings). The webapp relies heavily on this database working (user data, device data, prompts etc.).
      • app/
        Data generated from src/ on a ./Setup/setup.sh --frontend-app or a grunt deploy run. The need for a separate build vs source system is for the most part the application cache (appcache). See appcache is a douche. In order to work around all the caveats of the beloved (not) appcache, we need to for example change the filenames of views and scripts on each compile, generate a app/app.appcache manifest and more. This is all done in Gruntfile.js, see more details below. This build process can of course in addition do other useful things such as minifying the javascript code (which it does), and watching changes in the src/sass/app.scss file (grunt watch:sass). The server that serves the website should serve it from this folder (app/).
      • Gruntfile.js
        A file built almost entirely around the appcache. Handles cache breaking file names (appending a date), minification and more. Take care that as of date, if you add some files or modify paths in your application (which are not already added through the glob patterns), you have to add them to the appcache, this is the file to modify, if you want the application to work offline.

    Note: Since this was based on the angular-seed code, you can still run npm start if you want to run the application from the built in Angular local server (this feature is not actively tested, just happens to be there). So in addition to that, you could run the Flask application separate on a different port (although you would have to add CORS, and modify the BACKENDURL in src/app.js). It is probably simpler to use our apache server for the entire thing.

  • Local
    Code initially generated by the Setup/setup.sh script. Contains the apache and celery (QC) error logs at Log/. Also, as seen in Setup, contains the generated *.conf files which /etc/apache2/sites-enabled/datool.conf uses (Local/etc/apache2/vhosts/*.conf).

  • Setup
    Written to simplify the setup of Eyra. Most of the setup can be done by running setup.sh, either with the --all option or other specific ones (e.g. ./setup.sh --all --no-ap --no-mysqldb to run everything except the wifi access point and the database setup, or ./setup.sh --mysqldb to only run the database setup). ./setup.sh --all runs all components not put specifically as an external component (e.g. ext-kaldi).

    • If doing managed=false to managed=true doesn't work then also do the following:

      sudo rm /etc/hostapd/hostapd.conf

      sudo apt-get purge hostapd

      sudo rm /etc/dnsmasq.conf

      sudo apt-get purge dnsmasq

      Within /etc/network/interfaces file replace all the uncommented out lines with the default ethernet or wifi interfaces( names) and configurations, whether static or dynamic, your operating system uses. If you are running Debian 8 Jessie:

      # The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto wlan0
iface wlan0 inet dhcp

      Computers need a network card that can make an access point if you want to run ./setup.sh --ap. Also, in Setup/src/ap/default.conf change the settings as appropriate to work with the names of your drivers and network interface names.

      Then, restart your computer.

    The basic design is this:
    The setup.sh script contains a list of components, which correspond to folders in src (e.g. src/backend-qc which handles setting up the quality control). Each component can have

    • pip3 dependencies (pip3.deps)
    • apt-get dependencies (aptitude.deps)
    • A custom.deps.sh script for any additional dependencies.
    • Default configurations for variables to be replaced (default.conf)
    • Either in local files (files which end up in Local, listed in local.files) or
    • Global files (files changed on the system itself, listed in global.files).
    • Templates to create the local or global files using the variables in default.conf if necessary (tmpl/*)
    • A post_install.sh script. Run after everything else has been run (dependencies, file creation, etc.). You can put any component specific bash work here.

    As always, looking at the components already there will be a much better lesson than this documentation.

Description of individual Frontend services

  • androidRecording.service.js
    Service to handle recording through the Android WebView.
    Called in exactly the same way as recording.service.js, must therefore export same functions.
    Assumes interface from WebView, AndroidRecorder

  • authentication.service.js
    Right now pretty useless to have it as a special service (just uses $auth module directly), but if authentication scheme is ever changed or modified, it might be nice, so why not have it?

  • data.service.js
    Send data from one pageview to the next by injecting this service on both sides, and using get/set with strings as keys.
    This is used extensively, and should always be kept in mind when saving/retrieving variables that should last the duration of the app session or transferred between views.

  • delivery.service.js
    Handles http post and get requests to server. "Implements" the Client-Server API and also handles sending recs from local db when syncing.

  • evaluation.service.js
    Service to query and process prompts and recordings from server for evaluation.

  • localdb.service.js
    Handles local forage actions, saves recordings to local db.

    Stores indices of session objects in localDb/sessionIdxs.
    For example localDb/sessionIdxs = ['localDb/sessions/0', etc.]
    then the sessionObject:

    localDb/sessions/0 = {'metadata' : sessionData,  
                      'recordings' : [{'blobPath' : blobPath, 'title' : wavTitle.wav }, ...] }

    where blobPath is the localForage index of the stored blob (it seems like you can only store blobs as single blobs and not as part of an object when you store them through localForage) see issue: localForage/localForage#380
    blobPath = localDb/sessions/0/blobs/0 where blob id is same as recording index in array recordings.

  • localdbmisc.service.js
    Handles local forage actions regarding instructor, speaker and device setting along with other misc stuff.
    See Client-server API for format, but something like this:

    • Stores speakers thusly: 
      speakers/username = {'name':name, 'gender':gender, 'dob':dob [, 'height':height, 'deviceImei':imei, 'fullName':fullName, 'email':email]}
    • Stores instructors as: instructorId = 13
    • Stores devices as: 
      device = {'userAgent':'user agent string', 'imei':12363563456}

  • location.service.js
    Service to handle GPS location, basically a wrapper around navigator.geolocation.

  • logger.service.js
    Service to log errors/console output. Also saves it to localForage, with key logs as one big string.

  • notification.service.js
    Service to notify user (e.g. during recording) with for example a popup on a certain amount of prompts read.

  • myLocalForage.service.js
    Simple wrapper for the angular-localForage library.
    Does nothing except keep a single boolean whether there are any indexed db operations still ongoing.
    WARNING only implements the functions currently used in the app, if any other functions are used, they need to be added here manually.

  • qc.service.js
    Service to handle all processing realted to the QC. Querying the server, processing the information for example.

  • recording.service.js
    Service to handle recording, basically a wrapper around recorderjs library.

  • route.service.js
    Handles

    • Route errors in route resolves
    • Login reroutes

    e.g. when someone tries to go to the recording page without having either initialized the app or signed a participant agreement (if turned on).

  • session.service.js
    Service to handle some session logic, info about start of session and logic to assemble the session (from speaker, instructor, device, rec, token and all that info) and create the json object needed to send to server according to client server API.

  • token.service.js
    Service to query and process tokens from server.

  • utility.service.js
    Service with utility functions for the app, commonly abbreviated util. Also contains many configurable aspects, token count grabbed from server, frequency of QC querying etc., see also app.js.

  • volumeMeter.service.js
    Service for the volume meter, uses volume-meter.js. Code in part from https://github.com/cwilso/volume-meter

Some useful info

  • A very useful command to restart the Flask application (python backend) is sudo service apache2 restart.

  • Prompts and tokens are used interchangeably. Originally we called them tokens, but later decided to start calling them prompts.

  • Look at ClientServerAPI.md in the project root for the REST api which is used to transmit data between client and server. This document needs to be maintained if there are made modifications to the methods therein.

  • Logs are located in Local/Log. A typical use case would be to for example have a terminal open with a tail -f {error.log,celery.log} to watch the apache error log and the celery (QC) log respectively.

  • Running ./Setup/setup.sh --all can be dangerous, because it runs the --mysqldb command which deletes the entire database (not the recordings though). For this reason, you now have to manually confirm you want to delete the database if you run this. Still care should be taken when using --all, and for example, could run ./Setup/setup.sh --all --no-mysqldb to leave the database untouched. And of course, remember to backup your database to avoid disasters like this (see Backend/scripts/backup_db_and_recs.sh).

  • Running ./Setup/setup.sh --all or ./Setup/setup.sh --ap sets up a wireless access point on your computer. If this is not what you want (your wifi is disabled when this is done for example), a way to enable the wifi is the following:

    • sudo nano /etc/NetworkManager/NetworkManager.conf -> change managed=false to managed=true
    • sudo service network-manager restart

    Wifi should now work again.

  • Run webapp straight from Frontend/da-webapp/src:
    For development, it can be nice, not having to do a grunt deploy (with minification etc. and the time it takes). In which case, you can work straight from source, however some files need to be modified. These are:

    After making these modifications, you should be able to run application straight from src/. You will also probably want to run grunt watch:sass if you make any changes to the src/sass/app.scss file.

  • Monitoring apache (access the server-status page) can be done by navigating to /diagnostics-status (yoursite.com/diagnostics-status). You are prompted for a password on your first Setup/setup.sh --apache run. The username is admin. Number of useful things there, like requests per second, idle workers etc.

  • If you run into trouble getting data from phones to server (this happened with some older phones, and when server couldn't handle load), you could try making a Firebase account and submitting all the data to there aswell, you can see how we did it, you need to uncomment the async and firebase script libraries in index.html and the code at the top of services/delivery.service.js->submitRecordings.

  • You can see an example of converting data from another database/format to Eyra format in Backend/scripts/convert_to_eyra_database/malromur.

  • In order to collect data from people born before 2015, you need to change Frontend/da-webapp/src/json/speaker-info-format.json. Then grunt deploy from within the Frontend/da-webapp folder.

  • The max prompt length before the scrollbar comes into play is approximately 84 characters.

Maintaining code

Most of this should be self explanatory, but if you change code, remember to:

  • Write tests or make sure existing tests pass. (for now tests are seriously lacking, but you can run some Frontend tests with npm test and Backend tests in Backend/server-interface/test)
  • Change relevant comments.
  • Change relevant documentation (for example here in DEVELOPER.md and any other descriptors)
  • Rerun doctoc --notitle DEVELOPER.md if you modify this file's headings (manual for now). You can install doctoc by running sudo npm install -g doctoc.

Quality Control (QC)

NOTE currently, the frontend queries the QC as normal, but doesn't display anything it receives back.

The quality control is designed to process the recordings and try to improve the quality of gathered data by giving feedback to the user on the quality, allowing him to improve on the recordings he makes.

This QC uses Celery and a task chaining system to handle load and remain scalable (that's the idea anyway). By processing only a batch of recordings at a time, and then putting the continuation back on the queue as a task. Celery keeps the chain going, waiting for input from the speaker for a default of 15 minutes until it times out.
The QC is located at Backend/server-interface/qc.

See qc/celery_config.py for configurable parameters, such as how long a process sleeps on an idle task (no more recordings to be processed yet), and how long the timeout is.

Once the Celery chain for a speaker session has timed out, the QC reports are dumped on disk at /data/eyra/qc_reports or as specified in Backend/server-interface/qc/celery_config.py. As well as being saved in the Redis datastore (Redis is also used as a message broker for Celery).

Logging is done to Local/Log/celery.log. (be careful, still uses loglevel info (might want to change this for release), so the file could get big fast).

Firing up the QC

  • The QC needs Kaldi to be installed. This is done by running ./Setup/setup.sh --ext-kaldi. This could take some time (hrs).

  • If you also need the monophone models/tri models/etc for these modules, you can look at Backend/scripts/data_prep/run.sh. Here is an example of the commands needed to run to create the models/files for the Marosijo module:

    • Edit run.sh to use your lexicon and phonemes.txt.
    • ./run.sh

    The files for Marosijo should be located at local/marosijo.tgz after you run. Then you need to extract that and put it in a folder called qc/modules/MarosijoModule/local.

  • In addition, you need to obtain .scp and .ark files containing the decoded graphs (either by running e.g. qc/scripts/MarosijoGenGraphs.py or getting it elsewhere). Each decoding graph corresponds to a single prompt, and in this process you supply the prompts to the genGraphs scripts. It is important that the ids of the prompts supplied match the ids of the prompts in the database. These are then used by the Marosijo module to analyse the recording compared to each prompt. Generating these graphs takes a long time, and depends on the number/length of the prompt list. Look at qc/scripts/genGraphs.sh for parallelization of these.

  • Then, to restart Celery run ./Setup/setup.sh --backend-qc.

Selecting modules to use

In order to decide which QC modules to use, you need to modify config.py, both the imports and the activeModules dict to include the QC modules you want to use.
The default is only the Marosijo module.

If any modules are added/removed to/from activeModules, the setupActiveModules.py script needs to be run to add them to the relevant places, which is celery_handler.py because celery_handler.py needs to use its BaseTask to create its own processing task for each module. Format of adding modules to activeModules: mod=dict(name='UniqueNameModule', task='UniqueNameTask', processFn=qcProcSessionUniqueNameModule)

For example, I have a module TestModule, which is in TestModule.py Then I add TestModule=dict(name='TestModule', task='TestTask', processFn=qcProcSessionTestModule) to activeModules and add

try:
    # you need to manually add imports here
    from .celery_handler import qcProcSessionTestModule
except SystemError:
    # and here
    qcProcSessionTestModule = None

at the top.

Then the script will handle adding: from .celery_handler import qcProcSessionTestModule and adding the template code and replacing @celery.task(base=TestTask) and TestModule with your module in celery_handler.py.

Creating your own modules

To add your own QC module (lets call it New), you need to satisfy a couple of criteria;

  • Add a file modules/NewModule/NewModule.py.
  • Look at e.g. modules/TestModule/TestModule.py for reference, but you need to create a class, NewTask, which inherits from celery.Task and which will be used as a base Task for all subtasks using that QC module. This means, all subtasks have access to the data in the NewTask class (see celery docs for more details on this).
    • NewTask needs to be able to connect to the redis datastore, to modify the report for this QC module for each session. (with key report/NewModule/session_id or as specified in redis_layout.md) It is important that this report be JSON (double quotes people), use e.g. json.dumps before writing to redis datastore.
    • NewTask needs to define a method, processBatch which handles the processing that QC module needs to do. This method, takes as an argument a session id and indices (=[] when no new recordings are to be processed, in which case the function should return True) for recordings of that session to process in this batch (this list is stored in redis datastore by QC handler), and the task chaining which Celery uses, expects this processing function to work only on a small batch of recordings at a time (e.g. 5), so as to not take too long and be able to write intermediary results to redis to display on the client app. processBatch might take additional arguments, as specified in modules/TestModule/TestModule.py.
  • Modify the config.py script, as specified in the Selecting modules to use section.
  • Notes:
    • All files in modules/NewModule/local will be ignored by git as per .gitignore).

Running QC offline (post-processing of recordings)

The QC saves its reports on disk as well as in memory. This is saved to /data/eyra/qc_reports or as specified in Backend/server-interface/qc/celery_config.py.

It should be simple to test the QC on a session basis (if the reports don't already exist (they do if you had QC online while the recordings were made)). Simply create a script which queries the correct API endpoint (same as the clients do, see ClientServerAPI.md), i.e. /qc/report/session/<int:sessionId> for each session you want to generate a QC report for. Take special note though, that you have to routinely check the same sessions again to avoid a timeout (or change the session_timeout value in celery_config.py and restart celery (e.g. by running ./Setup/setup.sh --backend-qc)). The timeout is currently 15 minutes.
And so said script was made, see qc/scripts/runQCOffline.py. Run python3 runQCOffline.py --help for more options.
Look at qc/redis_layout.md to see how you can monitor the redis database during processing. For example, you can run redis-cli -n 1 --scan --pattern "*processing" to see which reports are being processed (or were being processed and encountered an error) and redis-cli -n 1 --scan --pattern "report*" to see which reports are in the redis database but haven't been dumped to disk (you can manually dump using qc/scripts/dumpCertainReports.py).

You'll want to modify qc/celery_config.py and set qc_offline_mode to True and you probably don't want to use the entire timeout mechanism (if you do, remember to requery the sessions to avoid the timeout), and can therefore set session_timeout=999999.
It might be better to work on a bigger batch size (modifiable in qc/celery_config.py). E.g. change batch_size to 500 and qc_big_batch_mode to True. Then you can skip requerying the sessions to avoid the timeout, and they should be done in big batches. An example script which seemed to perform well was this one (query all sessions sequentially and wait 7 minutes before querying the next one):

python3 runQCOffline.py --avoid_timeout 99999 --sleep_between 420;

This can obviously be different depending on your computer setup and data. Don't forget to set qc_big_batch_mode to True in qc/celery_config.py. That will ignore the timeout altogether, and dump the report on disk the moment it is ready.
In dire situations do a redis-cli -n 1 flushdb (or -n 0). And then restart Celery/apache. Or a computer restart to free memory.

There exist scripts to parse/process these QC dumps. See some useful scripts to this end in qc/scripts/process_qc_dumps And as an example, to create a sample of 200 recordings taken randomly from May and June along with its analysis from Marosijo see below. You then have to manually locate those recordings (hello find -name).

Example usage:

python3 parse_qc_dump.py /data/eyra/qc_reports/report/MarosijoModule Marosijo > qc_dump_combined.txt
./choose_qc_dump_combined.sh qc_dump_combined.txt 200 05 06

Existing modules

Marosijo module

An aligner which uses a phone bigram model to be able to insert phonemes between words in the prompt. Based on method in

[1] Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings (Vol. 2015-August, pp. 5206–5210). http://doi.org/10.1109/ICASSP.2015.7178964

Here is a bit taken from marosijo_make_utterance_fsts.sh:

G FST where P (i.e. <lm-fst>) is a phoneme bigram model and wi is the word sequence:

          _eps_   _eps_   _eps_   _eps_
         / w1  \ /  w2 \ /  w3 \ /  w4 \
       (1)---->(2)---->(3)---->(4)---->(5)
      /   \   /   \   /   \   /   \   /   \
      \_P_/   \_P_/   \_P_/   \_P_/   \_P_/

so the wis could be e.g. The quick fox jumped and a possible hypothesis would then be The quick p1 p2 p3 jumped where the pis would be e.g. phonemes representing pog meaning the aligner didn't get fox correctly.

Cleanup module

A simple aligner. It is considerably faster than Marosijo, both realtime and in creating the decoding graphs . Uses a small list of top words from the list of prompts and places higher probability on those words being recognized.

Setting this module up is similar to setting up Marosijo described in the Firing up the QC section.

Evaluation

A feature to allow users to evaluate (grade and comment) on the recording quality of specific recordings.

Located at yoursite.com/#/evaluation and with a link to it in the navbar (hamburger).

A special set "Random" is in place by default (a single record in evaluation_sets with special handling by the backend (i.e. serving random recordings)).

Usage

The grading system uses a 4 point scale. 1 is the worst possible grade and 4 is the best possible.

The basic steps when evaluating would be the following:

  • Listen
  • Comment (if necessary)
  • Grade

You listen to the utterance, if there was something wrong, you select what was wrong from the dropdown menu and finally you select a grade, which will immediately take you to the next prompt.

Notes:

  • You can tick the Autoplay checkbox if you want the next utterance to start playing automatically.
  • Clicking Skip will bring the next utterance without grading the current one.
  • The Undo button will allow you to go one and only one utterance back and re-grade.
  • Regarding the grading scale 1-4; of course these can be interpreted differently, but the main idea is that 1 would be what the evaluator sees as a useless recording, 4 would be a perfect recording (at least the entire prompt spoken correctly). If you are unsure, 2 would mean you think it is probably bad and 3 you think it is probably good.

Creating sets for evaluation

All you have to do is insert into evaluation_sets a label (eval_set) and a recordingId for all the recordings you want to be part of that set.

A 5 recording set example:

INSERT INTO evaluation_sets (eval_set, recordingId)
VALUES 
  ('example_set', 50),
  ('example_set', 51),
  ('example_set', 12),
  ('example_set', 5),
  ('example_set',196);