Skip to content
This repository was archived by the owner on Apr 14, 2025. It is now read-only.

Deep Dive | Templating

Jump to bottom
Smith, Tim edited this page Dec 21, 2016 · 6 revisions

Open Data Exporter uses doT to compile and render templates. doT is similar to handlebars or mustache in that it uses double curly braces ({{ }}) for expressions, but doT is much more powerful. doT allows not only variable evaluation, but also array iteration, variable assignment, and inline JavaScript function calls, all from within an expression.

Table of Contents

Variables and Data

Before covering the ins and outs of templating, it's important to first understand the variables and data used in templates.

The def object

The primary entry point for accessing any variables, data, or functions is the def object (def is for "definitions"). It contains the following standard properties:

  • now A moment.js object initialized to the local environment's current time and timezone when the job is started
  • data An object containing the responses from requests. The data from a request will generally be available at def.data.request_key where request_key is the key string as defined in the configuration's executionPlan property.
  • job The currently executing dereferenced job object from the config file
  • configuration The currently executing dereferenced configuration object from the config file
  • vars An object of key/value pairs that contains standard variables as well as any variables loaded from customData properties on config objects

def.vars - default properties

The def.vars object is initialized with several useful variables:

  • args An object of key/value pairs representing the command line parameters passed to node, in lower case. Flags are set as a boolean (e.g. using /debugApi results in "debugapi":true). Parameters are set as the expected key/value (e.g. using /logLevel=info results in "loglevel":"info").
  • date A moment.js object cloned from def.now, but converted to the timezone specified in the custom data variable timezoneOverride if a value was provided
  • interval The ISO-8601 duration used for calculating intervals. Defaults to 30 minutes with the value PT30M.

def.vars - custom data loading

When a job's configuration is executed, def.vars is populated with custom data. The custom data is loaded from customData properties on objects in the config file in the order below. Any duplicate keys will overwrite the previously set value.

  1. config.customData Custom data on the root of the config
  2. job.customData Custom data on the job
  3. configuration.customData Custom data on the configuration being processed
  4. configuration.executionPlan Custom data from each object is loaded in the order it is defined
  5. configuration.templates Custom data from each object is loaded in the order it is defined
  6. configuration.exports Custom data from each object is loaded in the order it is defined

def.vars - derived properties

After def.vars is initialized with default properties and custom data is loaded, the following properties are derived from properties set in def.vars:

  • Variables derived from defs.vars.date
    • startOfHour A moment.js object at the beginning of the current hour
    • previousStartOfHour A moment.js object one hour before the beginning of the current hour
    • startOfDay A moment.js object at the beginning of the current day
    • previousStartOfDay A moment.js object one day before the beginning of the current day
    • startOfWeek A moment.js object at the beginning of the current week
    • previousStartOfWeek A moment.js object one week before the beginning of the current week
    • startOfMonth A moment.js object at the beginning of the current month
    • previousStartOfMonth A moment.js object one month before the beginning of the current month
    • startOfQuarter A moment.js object at the beginning of the current quarter
    • previousStartOfQuarter A moment.js object one quarter before the beginning of the current quarter
    • startOfYear A moment.js object at the beginning of the current year
    • previousStartOfYear A moment.js object one year before the beginning of the current year
  • Variables derived from defs.vars.date and def.vars.interval
    • currentIntervalStart A moment.js object at the beginning of the current interval, calculated based on intervals always starting at the most recent midnight
    • previousIntervalStart A moment.js object one interval before the beginning of the current interval
    • currentInterval An ISO-8601 interval spanning the defined interval duration (_def.vars.interval) that encompasses the current time (def.vars.date)
    • previousInterval An ISO-8601 interval spanning the interval prior to the current interval

def.vars - special properties

The following custom data properties are explicitly used by the application:

  • interval Overrides the default interval (PT30M). Must be a valid ISO-8601 duration.
  • timezoneOverride Overrides the default timezone (environment context) for all date objects. Timezones are parsed using moment-timezone and should be in a valid tz database format (e.g America/Indianapolis). Numeric offsets are not allowed because they do not reflect DST observance.
  • request Do not attempt to set this. When executing transforms during the request templating process, def.vars.request is set to the request body object and is cleared afterwards.

External Modules (JavaScript Functions)

For templates to use functions, they must be loaded into the def object. This is done by loading the files listed in a configuration's externalModules object as node modules into the def object. They are loaded with the variable name matching the key in the configuration.

Given this configuration:

"externalModules": {
  "exampleFunctions": "./extensions/examples/examples.js"
}

The functions exported by the module in examples.js can now be accessed via def.exampleFunctions. If the module defines a function named helloWorld, it can be invoked as def.exampleFunctions.helloWorld().

Additionally, all .js files found in ./extensions/standard are automatically loaded as modules with names matching the filenames.

Tips

  • The module must export an anonymous object (module.exports = new MyModule();) as opposed to an anonymous prototype (module.exports = MyModule;).
  • The location of the module must have access to all required node dependencies. The recommended pattern is to place them in the src/extensions/<yourmodule>/ directory and add your dependencies to src/package.json. This will give your modules access to any modules defined in package.json and will automatically install them when running npm install.
  • Don't include your module with a name that will conflict with anything else. For example, vars, data, job, and configuration are all bad choices and will break the application.

Compile-time vs Evaluation-time vars

The template execution process is done in two steps.

First, the template is compiled by parsing the template into a JavaScript function that can be executed. During the compilation, the only valid root variable is the def object.

Second, the resulting JavaScript function from the compilation process is executed. During the execution stage, the only valid root variable is it.

The standard expectation is that def and it can be used interchangeably and will be the exact same object. The templates {{#def.data.name}} and {{#it.data.name}} will result in the exact same output.

There are some exceptions, however:

  • Collection iteration in templates can only be done on the it variable. Attempts to iterate a collection using the dev variable will cause a compilation error to be thrown when compiling the template.
  • When executing repeating requests, def will contain the standard object, but it will contain the current collection member that is in context for the request.

Template Syntax

The following sections will assume that the def object is defined as:

{
  "vars": {
    "myInt": 5,
    "myString": "a value"
  },
  "data": {
    "name": "Steve",
    "age": 25,
    "address": {
      "street": "123 Strasse St.",
      "city": "Denver",
      "state": "CO",
      "zip": "80123"
    },
    "cars": [
      {
        "make": "Ford",
        "model": "Mustang"
      },
      {
        "make": "Subaru",
        "model": "WRX"
      }
    ]
  },
  "helpers": {
    "addNumbers": function(num1, num2) { return num1 + num2; },
    "flattenAddress": function(data) { data.fullAddress = data.address.street + ', ' + data.address.city + ', ' + data.address.state + ' ' + data.address.zip; }
  }
}

Variable Evaluation

The most basic operation is to evaluate a variable. To do this, use the syntax {{# varname }} where varname is the name of the variable to evaluate. This can include subobjects like def.vars.myString.

This template:

{{#def.data.name}} is {{#def.data.age}} years old and lives on {{#def.data.address.street}}.

will evaluate to:

Steve is 25 years old and lives on 123 Strasse St..

Calling functions

Functions that have been loaded from external modules can be executed in a template. The value returned from the function will be inserted into the template. The value should be a string or other primitive type. The value will have .toString() invoked on it and will insert "[object Object]" into the template if the value isn't a primitive type.

This template:

{{#def.data.name}} is {{#def.data.age}} years old now. In {{#def.vars.myInt}} years, he will be {{#def.helpers.addNumbers(def.data.age, def.vars.myInt)}}.

will evaluate to:

Steve is 25 years old now. In 5 years he will be 30.

Assigning the result of a function to a variable

In some cases it may be useful to assign the output of a function to a variable. When doing so, the function may return any type and the value returned will be stored in the variable, even if an object is returned.

This template:

{{#(def.data.futureAge=def.helpers.addNumbers(def.data.age, def.vars.myInt)) && ""}}

will evaluate to an empty string and will not display any data. However, after executing this expression, this template:

{{#def.data.name}} will be {{#def.data.futureAge}} years old in {{#def.vars.myInt}} years.

will evaluate to:

Steve will be 30 years old in 5 years.

The syntax for variable assignment can be genericised as {{#(expression) && ""}}.

Using functions to manipulate objects

All objects passed into functions are passed by reference and can be manipulated without a need to return any value. This use of functions is particularly useful for creating transforms.

This template:

{{#def.helpers.flattenAddress(def.data)}}

will evaluate to an empty string since no value is returned. However, def.data.fullAddress will now have a value. After executing the template above, this template:

{{#def.data.name}} lives at {{#def.data.fullAddress}}.

will evaluate to:

Steve lives at 123 Strasse St., Denver, CO 80123.

Iterating collections

Collections can be iterated to execute a template for each member of the collection. The template for iteration is everything that is contained between the open and closing expressions for an array.

To mark the beginning of an iteration template, use the format: {{~collection :varname}}. To mark the end of the iteration template, use the format: {{~}}. Within the template, the current item in the collection can be accessed with the format: {{=varname}}.

This template:

{{#def.data.name}} owns these cars: {{~it.data.cars :car}} a {{=car.make}} {{=car.model}}, {{~}}.

will evaluate to:

Steve owns these cars: a Ford Mustang, a Subaru WRX, .

Tips

  • The output has a trailing ", " at the end. Iterating data sets like this may be better handled by a function because the function can trim trailing characters or execute different logic per array member. Template collection iteration is well suited for use cases where each iteration ends in a newline or there are no trailing characters used.
  • Iterating collections is only possible using the it variable. For more information, see the section Compile-time vs Evaluation-time vars above.
Clone this wiki locally