HullConnector

Parameters

HullClient HullClient
options Object (optional, default {})
- options.hostSecret string? secret to sign req.hull.token
- options.port (Number | string)? port on which expressjs application should be started
- options.clientConfig Object? additional HullClient configuration (optional, default {})
- options.instrumentation Object? override default InstrumentationAgent
- options.cache Object? override default CacheAgent
- options.queue Object? override default QueueAgent
- options.connectorName string? force connector name - if not provided will be taken from manifest.json
- options.skipSignatureValidation boolean? skip signature validation on notifications (for testing only)
- options.timeout (number | string)? global HTTP server timeout - format is parsed by ms npm package
- options.segmentFilterSetting

setupApp

This method applies all features of Hull.Connector to the provided application:

serving /manifest.json, /readme and / endpoints
serving static assets from /dist and /assets directiories
rendering /views/*.html files with ejs renderer
timeouting all requests after 25 seconds
adding Newrelic and Sentry instrumentation
initiating the wole Context Object
handling the hullToken parameter in a default way

Parameters

app express expressjs application

Returns express expressjs application

startApp

This is a supplement method which calls app.listen internally and also terminates instrumentation of the application calls.

Parameters

app express expressjs application

Returns http.Server

Errors

General utilities

ConfigurationError

Extends TransientError

This is an error related to wrong connector configuration. It's a transient error, but it makes sense to retry the payload only after the connector settings update.

Parameters

message string
extra Object

LogicError

Extends Error

This is an error which should be handled by the connector implementation itself.

Rejecting or throwing this error without try/catch block will be treated as unhandled error.

Parameters

message string
action string
payload any

Examples

function validationFunction() {
  throw new LogicError("Validation error", { action: "validation", payload: });
}

RateLimitError

Extends TransientError

This is a subclass of TransientError. It have similar nature but it's very common during connector operations so it's treated in a separate class. Usually connector is able to tell more about when exactly the rate limit error will be gone to optimize retry strategy.

Parameters

message string
extra Object

RecoverableError

Extends TransientError

This error means that 3rd party API resources is out of sync comparing to Hull organization state. For example customer by accident removed a resource which we use to express segment information (for example user tags, user sub lists etc.) So this is a TransientError which could be retried after forcing "reconciliation" operation (which should recreate missing resource)

Parameters

message string
extra Object

TransientError

Extends Error

This is a transient error related to either connectivity issues or temporary 3rd party API unavailability.

When using superagentErrorPlugin it's returned by some errors out-of-the-box.

Parameters

message string
extra Object

Context

An object that's available in all action handlers and routers as req.hull. It's a set of parameters and modules to work in the context of current organization and connector instance.

helpers

This is a set of additional helper functions being exposed at req.hull.helpers. They allow to perform common operation in the context of current request. They are similar o req.hull.client.utils, but operate at higher level, ensure good practises and should be used in the first place before falling back to raw utils.

handleExtract

Helper function to handle JSON extract sent to batch endpoint

Parameters

ctx Object Hull request context
options Object
- options.body Object request body object (req.body)
- options.batchSize Object size of the chunk we want to pass to handler
- options.handler Function callback returning a Promise (will be called with array of elements)
- options.onResponse Function callback called on successful inital response
- options.onError Function callback called during error

Returns Promise

requestExtract

This is a method to request an extract of user base to be sent back to the Connector to a selected path which should be handled by notifHandler.

Parameters

ctx Object Hull request context
options Object (optional, default {})
- options.segment Object (optional, default null)
- options.format Object (optional, default json)
- options.path Object (optional, default /batch)
- options.fields Object (optional, default [])
- options.additionalQuery Object (optional, default {})

Examples

req.hull.helpers.requestExtract({ segment = null, path, fields = [], additionalQuery = {} });

Returns Promise

updateSettings

Allows to update selected settings of the ship private_settings object. This is a wrapper over hullClient.utils.settings.update() call. On top of that it makes sure that the current context ship object is updated, and the ship cache is refreshed. It will emit ship:update notify event.

Parameters

ctx Object The Context Object
newSettings Object settings to update

Examples

req.hull.helpers.updateSettings({ newSettings });

Returns Promise

cache

Cache available as req.hull.cache object. This class is being intiated and added to Context Object by QueueAgent. If you want to customize cache behavior (for example ttl, storage etc.) please @see Infra.QueueAgent

wrap

See: https://github.com/BryanDonovan/node-cache-manager#overview

Hull client calls which fetch ship settings could be wrapped with this method to cache the results

Parameters

key string
cb Function callback which Promised result would be cached
options Object?

Returns Promise

set

Saves ship data to the cache

Parameters

key string
value mixed
options Object?

Returns Promise

get

Returns cached information

Parameters

key string

Returns Promise

del

Clears the ship cache. Since Redis stores doesn't return promise for this method, it passes a callback to get a Promise

Parameters

key string

Returns any Promise

metric

Metric agent available as req.hull.metric object. This class is being initiated by InstrumentationAgent. If you want to change or override metrics behavior please @see Infra.InstrumentationAgent

Examples

req.hull.metric.value("metricName", metricValue = 1);
req.hull.metric.increment("metricName", incrementValue = 1); // increments the metric value
req.hull.metric.event("eventName", { text = "", properties = {} });

value

Sets metric value for gauge metric

Parameters

metric string metric name
value number metric value (optional, default 1)
additionalTags Array additional tags in form of ["tag_name:tag_value"] (optional, default [])

Returns mixed

increment

Increments value of selected metric

Parameters

metric string metric metric name
value number value which we should increment metric by (optional, default 1)
additionalTags Array additional tags in form of ["tag_name:tag_value"] (optional, default [])

Returns mixed

event

Parameters

options Object
- options.title string
- options.text string (optional, default "")
- options.properties Object (optional, default {})

Returns mixed

enqueue

Parameters

queueAdapter Object adapter to run - when using this function in Context this param is bound
ctx Context Hull Context Object - when using this function in Context this param is bound
jobName string name of specific job to execute
jobPayload Object the payload of the job
options Object (optional, default {})
- options.ttl number? job producers can set an expiry value for the time their job can live in active state, so that if workers didn't reply in timely fashion, Kue will fail it with TTL exceeded error message preventing that job from being stuck in active state and spoiling concurrency.
- options.delay number? delayed jobs may be scheduled to be queued for an arbitrary distance in time by invoking the .delay(ms) method, passing the number of milliseconds relative to now. Alternatively, you can pass a JavaScript Date object with a specific time in the future. This automatically flags the Job as "delayed".
- options.queueName string? when you start worker with a different queue name, you can explicitly set it here to queue specific jobs to that queue
- options.priority (number | string)? you can use this param to specify priority of job

Examples

// app is Hull.Connector wrapped expressjs app
app.get((req, res) => {
  req.hull.enqueue("jobName", { payload: "to-work" })
    .then(() => {
      res.end("ok");
    });
});

Returns Promise which is resolved when job is successfully enqueued

Meta

deprecated: internal connector queue is considered an antipattern, this function is kept only for backward compatiblity

Infra

Production ready connectors need some infrastructure modules to support their operation, allow instrumentation, queueing and caching. The Hull.Connector comes with default settings, but also allows to initiate them and set a custom configuration:

Examples

const instrumentation = new Instrumentation();
const cache = new Cache();
const queue = new Queue();

const connector = new Hull.Connector({ instrumentation, cache, queue });

CacheAgent

This is a wrapper over https://github.com/BryanDonovan/node-cache-manager to manage ship cache storage. It is responsible for handling cache key for every ship.

By default it comes with the basic in-memory store, but in case of distributed connectors being run in multiple processes for reliable operation a shared cache solution should be used. The Cache module internally uses node-cache-manager, so any of it's compatibile store like redis or memcache could be used:

The cache instance also exposes contextMiddleware whch adds req.hull.cache to store the ship and segments information in the cache to not fetch it for every request. The req.hull.cache is automatically picked and used by the Hull.Middleware and segmentsMiddleware.

The req.hull.cache can be used by the connector developer for any other caching purposes:

ctx.cache.get('object_name');
ctx.cache.set('object_name', object_value);
ctx.cache.wrap('object_name', () => {
  return Promise.resolve(object_value);
});

There are two object names which are reserved and cannot be used here:

any ship id

"segments"

IMPORTANT internal caching of ctx.ship object is refreshed on ship:update notifications, if the connector doesn't subscribe for notification at all the cache won't be refreshed automatically. In such case disable caching, set short TTL or add notifHandler

Parameters

options Object passed to node-cache-manager (optional, default {})

Examples

const redisStore = require("cache-manager-redis");
const { Cache } = require("hull/lib/infra");

const cache = new Cache({
  store: redisStore,
  url: 'redis://:XXXX@localhost:6379/0?ttl=600'
});

const connector = new Hull.Connector({ cache });

InstrumentationAgent

It automatically sends data to DataDog, Sentry and Newrelic if appropriate ENV VARS are set:

NEW_RELIC_LICENSE_KEY
DATADOG_API_KEY
SENTRY_URL

It also exposes the contextMiddleware which adds req.hull.metric agent to add custom metrics to the ship. Right now it doesn't take any custom options, but it's showed here for the sake of completeness.

Parameters

options (optional, default {})

Examples

const { Instrumentation } = require("hull/lib/infra");

const instrumentation = new Instrumentation();

const connector = new Connector.App({ instrumentation });

QueueAgent

By default it's initiated inside Hull.Connector as a very simplistic in-memory queue, but in case of production grade needs, it comes with a Kue or Bull adapters which you can initiate in a following way:

Options from the constructor of the BullAdapter or KueAdapter are passed directly to the internal init method and can be set with following parameters:

https://github.com/Automattic/kue#redis-connection-settings https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queue

The queue instance has a contextMiddleware method which adds req.hull.enqueue method to queue jobs - this is done automatically by Hull.Connector().setupApp(app):

req.hull.enqueue((jobName = ''), (jobPayload = {}), (options = {}));

By default the job will be retried 3 times and the payload would be removed from queue after successfull completion.

Then the handlers to work on a specific jobs is defined in following way:

connector.worker({
  jobsName: (ctx, jobPayload) => {
    // process Payload
    // this === job (kue job object)
    // return Promise
  }
});
connector.startWorker();

Parameters

adapter Object

Examples

```javascript
const { Queue } = require("hull/lib/infra");
const BullAdapter = require("hull/lib/infra/queue/adapter/bull"); // or KueAdapter

const queueAdapter = new BullAdapter(options);
const queue = new Queue(queueAdapter);

const connector = new Hull.Connector({ queue });
```

Meta

deprecated: internal connector queue is considered an antipattern, this class is kept only for backward compatiblity

Hull.Middleware

This middleware standardizes the instantiation of a Hull Client in the context of authorized HTTP request. It also fetches the entire ship's configuration.

Parameters

HullClient HullClient Hull Client - the version exposed by this library comes with HullClient argument bound
options Object
- options.hostSecret string The ship hosted secret - consider this as a private key which is used to encrypt and decrypt req.hull.token. The token is useful for exposing it outside the Connector <-> Hull Platform communication. For example the OAuth flow or webhooks. Thanks to the encryption no 3rd party will get access to Hull Platform credentials.
- options.clientConfig Object Additional config which will be passed to the new instance of Hull Client (optional, default {})

Returns Function

Utils

General utilities

notifHandler

NotifHandler is a packaged solution to receive User and Segment Notifications from Hull. It's built to be used as an express route. Hull will receive notifications if your ship's manifest.json exposes a subscriptions key:

Note : The Smart notifier is the newer, more powerful way to handle data flows. We recommend using it instead of the NotifHandler. This handler is there to support Batch extracts.

{
  "subscriptions": [{ "url": "/notify" }]
}

Parameters

params Object
- params.handlers Object
- params.onSubscribe Function?
- params.options Object?
  - params.options.maxSize number? the size of users/account batch chunk
  - params.options.maxTime number? time waited to capture users/account up to maxSize
  - params.options.segmentFilterSetting string? setting from connector's private_settings to mark users as whitelisted
  - params.options.groupTraits boolean (optional, default false)
- params.userHandlerOptions Object? deprecated

Examples

import { notifHandler } from "hull/lib/utils";
const app = express();

const handler = NotifHandler({
  options: {
    groupTraits: true, // groups traits as in below examples
    maxSize: 6,
    maxTime: 10000,
    segmentFilterSetting: "synchronized_segments"
  },
  onSubscribe() {} // called when a new subscription is installed
  handlers: {
    "ship:update": function(ctx, message) {},
    "segment:update": function(ctx, message) {},
    "segment:delete": function(ctx, message) {},
    "account:update": function(ctx, message) {},
    "user:update": function(ctx, messages = []) {
      console.log('Event Handler here', ctx, messages);
      // ctx: Context Object
      // messages: [{
      //   user: { id: '123', ... },
      //   segments: [{}],
      //   changes: {},
      //   events: [{}, {}]
      //   matchesFilter: true | false
      // }]
    }
  }
})

connector.setupApp(app);
app.use('/notify', handler);

Returns Function expressjs router

Meta

deprecated: use smartNotifierHandler instead, this module is kept for backward compatibility

oAuthHandler

OAuthHandler is a packaged authentication handler using Passport. You give it the right parameters, it handles the entire auth scenario for you.

It exposes hooks to check if the ship is Set up correctly, inject additional parameters during login, and save the returned settings during callback.

To make it working in Hull dashboard set following line in manifest.json file:

{
  "admin": "/auth/"
}

For example of the notifications payload see details

Parameters

options Object
- options.name string The name displayed to the User in the various screens.
- options.tokenInUrl boolean Some services (like Stripe) require an exact URL match. Some others (like Hubspot) don't pass the state back on the other hand. Setting this flag to false (default: true) removes the token Querystring parameter in the URL to only rely on the state param.
- options.isSetup Function A method returning a Promise, resolved if the ship is correctly setup, or rejected if it needs to display the Login screen. Lets you define in the Ship the name of the parameters you need to check for. You can return parameters in the Promise resolve and reject methods, that will be passed to the view. This lets you display status and show buttons and more to the customer
- options.onAuthorize Function A method returning a Promise, resolved when complete. Best used to save tokens and continue the sequence once saved.
- options.onLogin Function A method returning a Promise, resolved when ready. Best used to process form parameters, and place them in req.authParams to be submitted to the Login sequence. Useful to add strategy-specific parameters, such as a portal ID for Hubspot for instance.
- options.Strategy Function A Passport Strategy.
- options.views Object Required, A hash of view files for the different screens: login, home, failure, success
- options.options Object Hash passed to Passport to configure the OAuth Strategy. (See Passport OAuth Configuration)

Examples

const { oAuthHandler } = require("hull/lib/utils");
const { Strategy as HubspotStrategy } = require("passport-hubspot");

const app = express();

app.use(
  '/auth',
  oAuthHandler({
    name: 'Hubspot',
    tokenInUrl: true,
    Strategy: HubspotStrategy,
    options: {
      clientID: 'xxxxxxxxx',
      clientSecret: 'xxxxxxxxx', //Client Secret
      scope: ['offline', 'contacts-rw', 'events-rw'] //App Scope
    },
    isSetup(req) {
      if (!!req.query.reset) return Promise.reject();
      const { token } = req.hull.ship.private_settings || {};
      return !!token
      ? Promise.resolve({ valid: true, total: 2 })
      : Promise.reject({ valid: false, total: 0 });
    },
    onLogin: req => {
      req.authParams = { ...req.body, ...req.query };
      return req.hull.client.updateSettings({
        portalId: req.authParams.portalId
      });
    },
    onAuthorize: req => {
      const { refreshToken, accessToken } = req.account || {};
      return req.hull.client.updateSettings({
        refresh_token: refreshToken,
        token: accessToken
      });
    },
    views: {
      login: 'login.html',
      home: 'home.html',
      failure: 'failure.html',
      success: 'success.html'
    }
  })
);

//each view will receive the following data:
{
  name: "The name passed as handler",
  urls: {
    login: '/auth/login',
    success: '/auth/success',
    failure: '/auth/failure',
    home: '/auth/home',
  },
  ship: ship //The entire Ship instance's config
}

Returns Function OAuth handler to use with expressjs

smartNotifierHandler

smartNotifierHandler is a next generation notifHandler cooperating with our internal notification tool. It handles Backpressure, throttling and retries for you and lets you adapt to any external rate limiting pattern.

To enable the smartNotifier for a connector, the smart-notifier tag should be present in manifest.json file. Otherwise, regular, unthrottled notifications will be sent without the possibility of flow control.

{
  "tags": ["smart-notifier"],
  "subscriptions": [
    {
      "url": "/notify"
    }
  ]
}

When performing operations on notification you can set FlowControl settings using ctx.smartNotifierResponse helper.

Parameters

params Object
- params.handlers Object
- params.options Object?
  - params.options.maxSize number? the size of users/account batch chunk
  - params.options.maxTime number? time waited to capture users/account up to maxSize
  - params.options.segmentFilterSetting string? setting from connector's private_settings to mark users as whitelisted
  - params.options.groupTraits boolean (optional, default false)
- params.userHandlerOptions Object? deprecated

Examples

const { smartNotifierHandler } = require("hull/lib/utils");
const app = express();

const handler = smartNotifierHandler({
  handlers: {
    'ship:update': function(ctx, messages = []) {},
    'segment:update': function(ctx, messages = []) {},
    'segment:delete': function(ctx, messages = []) {},
    'account:update': function(ctx, messages = []) {},
    'user:update': function(ctx, messages = []) {
      console.log('Event Handler here', ctx, messages);
      // ctx: Context Object
      // messages: [{
      //   user: { id: '123', ... },
      //   segments: [{}],
      //   changes: {},
      //   events: [{}, {}]
      //   matchesFilter: true | false
      // }]
      // more about `smartNotifierResponse` below
      ctx.smartNotifierResponse.setFlowControl({
        type: 'next',
        size: 100,
        in: 5000
      });
      return Promise.resolve();
    }
  },
  options: {
    groupTraits: false
  }
});

connector.setupApp(app);
app.use('/notify', handler);

Returns Function expressjs router

superagentErrorPlugin

This is a general error handling SuperAgent plugin.

It changes default superagent retry strategy to rerun the query only on transient connectivity issues (ECONNRESET, ETIMEDOUT, EADDRINFO, ESOCKETTIMEDOUT, ECONNABORTED). So any of those errors will be retried according to retries option (defaults to 2).

If the retry fails again due to any of those errors the SuperAgent Promise will be rejected with special error class TransientError to distinguish between logical errors and flaky connection issues.

In case of any other request the plugin applies simple error handling strategy: every non 2xx or 3xx response is treated as an error and the promise will be rejected. Every connector ServiceClient should apply it's own error handling strategy by overriding ok handler.

Parameters

options Object (optional, default {})
- options.retries Number? Number of retries
- options.timeout Number? Timeout for request

Examples

superagent.get("http://test/test")
  .use(superagentErrorPlugin())
  .ok((res) => {
    if (res.status === 401) {
      throw new ConfigurationError();
    }
    if (res.status === 429) {
      throw new RateLimitError();
    }
    return true;
  })
  .catch((error) => {
    // error.constructor.name can be ConfigurationError, RateLimitError coming from `ok` handler above
    // or TransientError coming from logic applied by `superagentErrorPlugin`
  })

Returns Function function to use as superagent plugin

superagentInstrumentationPlugin

This plugin takes client.logger and metric params from the Context Object and logs following log line:

ship.service_api.request with params:
- url - the original url passed to agent (use with superagentUrlTemplatePlugin)
- responseTime - full response time in ms
- method - HTTP verb
- status - response status code
- vars - when using superagentUrlTemplatePlugin it will contain all provided variables

The plugin also issue a metric with the same name ship.service_api.request.

Parameters

options Object
- options.logger Object Logger from HullClient
- options.metric Object Metric from Hull.Connector

Examples

const superagent = require('superagent');
const { superagentInstrumentationPlugin } = require('hull/lib/utils');

// const ctx is a Context Object here

const agent = superagent
.agent()
.use(
  urlTemplatePlugin({
    defaultVariable: 'mainVariable'
  })
)
.use(
  superagentInstrumentationPlugin({
    logger: ctx.client.logger,
    metric: ctx.metric
  })
);

agent
.get('https://api.url/{{defaultVariable}}/resource/{{resourceId}}')
.tmplVar({
  resourceId: 123
})
.then(res => {
  assert(res.request.url === 'https://api.url/mainVariable/resource/123');
});

> Above code will produce following log line:
```sh
connector.service_api.call {
  responseTime: 880.502444,
  method: 'GET',
  url: 'https://api.url/{{defaultVariable}}/resource/{{resourceId}}',
  status: 200
}
```

> and following metrics:

```javascript
- `ship.service_api.call` - should be migrated to `connector.service_api.call`
- `connector.service_api.responseTime`
```

Returns Function function to use as superagent plugin

superagentUrlTemplatePlugin

This plugin allows to pass generic url with variables - this allows better instrumentation and logging on the same REST API endpoint when resource ids varies.

Parameters

defaults Object default template variable

Examples

const superagent = require('superagent');
const { superagentUrlTemplatePlugin } = require('hull/lib/utils');

const agent = superagent.agent().use(
  urlTemplatePlugin({
    defaultVariable: 'mainVariable'
  })
);

agent
.get('https://api.url/{{defaultVariable}}/resource/{{resourceId}}')
.tmplVar({
  resourceId: 123
})
.then(res => {
  assert(res.request.url === 'https://api.url/mainVariable/resource/123');
});

Returns Function function to use as superagent plugin

Files

API.md

Latest commit

History

API.md

File metadata and controls

HullConnector

Parameters

setupApp

Parameters

startApp

Parameters

Errors

ConfigurationError

Parameters

LogicError

Parameters

Examples

RateLimitError

Parameters

RecoverableError

Parameters

TransientError

Parameters

Context

helpers

handleExtract

Parameters

requestExtract

Parameters

Examples

updateSettings

Parameters

Examples

cache

wrap

Parameters

set

Parameters

get

Parameters

del

Parameters

metric

Examples

value

Parameters

increment

Parameters

event

Parameters

enqueue

Parameters

Examples

Infra

Examples

CacheAgent

Parameters

Examples

InstrumentationAgent

Parameters

Examples

QueueAgent

Parameters

Examples

Hull.Middleware

Parameters

Utils

notifHandler

Parameters

Examples

oAuthHandler

Parameters

Examples

smartNotifierHandler

Parameters

Examples

superagentErrorPlugin

Parameters

Examples