Overview

Hull Client

A low level Hull Platform API client. Refer to it's documentation for more details

const hullClient = new Hull.Client({ configuration });

This is an example of the bare bones API client. Please refer to it's own Github repository for documentation.

Hull Middleware

app.use(Hull.Middleware({ configuration }));

A bridge between Hull Client and a NodeJS HTTP application (e.g. express) which initializes HullClient a context for every HTTP request. See example usage below. A standalone usage is possible (it's a strandard ExpressJS middleware), but if there is no specific reason to do so, the recommended way of building connectors is Hull Connector.

Hull Connector

const connector = new Hull.Connector({ configuration });

A complete toolkit which is created next to ExpressJS server instance. Includes Hull Middleware and a set of official patterns to build highly scalable and efficient Connectors.

To get started see few chapters of this README first:

start with Initialization and Setup Helpers
then have a quick look what you hava available in Context Object
proceed to Incoming data flow or Outgoing data flow depending on your use case

Hull.Middleware

Example usage

const Hull = require("hull");
const express = require("express");

const app = express();
app.use(Hull.Middleware({ hostSecret: "secret" }));
app.post("/show-segments", (req, res) => {
  req.hull.client.get("/segments").then(segments => {
    res.json(segments);
  });
});

This middleware standardizes the instantiation of a Hull Client in the context of authorized HTTP request. It also fetches the entire Connector's configuration. As a result it's responsible for creating and exposing a Context Object, another important part is how this middleware decide where to look for configuration settings (connector ID, SECRET and ORGANIZATION) which then are applied to HullClient, for details please refer to configuration resolve strategy.

For configuration options refer to API REFERENCE.

Hull.Connector

This is the smallest possible Nodejs connector implementation:

const app = express();
app.get("/manifest.json", serveTheManifestJson);
app.listen(port);

As you can see connector is a simple HTTP application served from public address. It can be implemented in any way and in any technological stack as long as it implements the same API. You can find more details on connector's structure here.

Initialization

const Hull = require("hull");

const connector = new Hull.Connector({
  port: 1234, // port to start express app on
  hostSecret: "secret", // a secret generated random string used as a private key
});

This is the instance of the Connector module which exposes a set of utilities which can be applied to the main express app. All configuration options are listen in API REFERENCE

The utilities and special middlewares can be taken one-by-one from the library and applied to the application manually, but to make the whole process easier, there are two helper methods that set everything up for you:

Setup Helpers

const express = require("express");
const Hull = require("hull");

const app = express();
const connector = new Hull.Connector({ hostSecret, port });

connector.setupApp(app); // apply connector related features to the application
app.post("/fetch-all", (req, res) => {
  // req.hull is the full Context Object!
  req.hull.client.get("/segments")
    .then((segments) => {
      res.json(segments);
    });
});
connector.startApp(app); // apply termination middlewares and internally calls `app.listen`

Setup Helpers are two high-level methods exposed by initialized Connector instances to apply custom middlewares to the Express application. Those middlewares enrich the request object with full Context Object.

To get more details on how those helpers methods work please see API REFERENCE

Context Object

Hull.Connector apply multiple middlewares to the request handler, including Hull.Middleware. The result is a Context Object that's available in all action handlers and routers as req.hull. It's a set of parameters and modules to work in the context of current organization and connector instance. This Context is divided into a base set by Hull.Middleware (if you use it standalone) and an extended set applied when using Hull.Connector and helpers method described above.

Here is the base structure of the Context Object (we also provide Flow type for this object here).

{
  // set by Hull.Middleware
  requestId: "",
  config: {},
  token: "",
  client: { // Instance of "new Hull.Client()"
    logger: {},
  },
  ship: {
    //The values for the settings defined in the Connector's settings tab
    private_settings: {},
    settings: {}
  },
  hostname: req.hostname,
  options: req.query + req.body,

  // set by Hull.Connector
  connectorConfig: {},
  segments: [],
  users_segments: [],
  accounts_segments: [],

  cache: {},
  enqueue: () => {},
  metric: {},
  helpers: {},
  service: {},
  message: {},
  notification: {}
  smartNotifierResponse: {}
}

Base Context - set by Hull.Middleware

requestId

unique identifier for a specific request. Will be used to enrich the context of all the logs emitted by the the Hull.Client logger. This value is automatically added by the notifHandler and smartNotifierHandler with the SNS MessageId or SmartNotifier notification_id.

config

an object carrying id, secret and organization. You can setup it prior to the Hull Middleware execution (via custom middleware) to override default configuration strategy.

token

an encrypted version of configuration. If it's already set in the request, Hull Middleware will try to decrypt it and get the configuration from it. If it's not available at the beginning and middleware resolved the configuration from other sources it will encrypt it and set req.hull.token value.

When the connector needs to send the information outside the Hull ecosystem it has to use the token, not to expose the raw credentials. The usual places where this happens are:

dashboard links
oAuth flow (callback url)
external incoming webhooks

client

Hull API client initialized to work with current organization and connector.

ship

ship object with manifest information and private_settings fetched from the Hull Platform. ship is the legacy name for Connectors.

hostname

Hostname of the current request. Since the connector are stateless services this information allows the connector to know it's public address.

options

Is the object including data from query and body of the request

Extended Context - set by `Hull.Connector`

connectorConfig

Hash with connector settings, details in Hull.Connector constructor reference.

segments

[
  {
    "name": "Segment name",
    "id": "123abc"
  }
]

An array of segments defined at the organization, it's being automatically exposed to the context object. The segment flow type is specified here.

users_segments param is alias to segments and accounts_segments exposes list of segments for accounts.

cache

Since every connector can possibly work on high volumes of data performing and handling big number of requests. Internally the cache is picked by the Hull Middleware to store the ship object and by segmentsMiddleware to store segments list. The cache can be also used for other purposes, e.g. for minimizing the External API calls. Caching Module is exposing three public methods:

ctx.cache.get("object_name");
ctx.cache.set("object_name", objectValue);
ctx.cache.wrap("object_name", (objectValue) => {
  return Promise.resolve(objectValue);
});

Full API reference

enqueue

This is generally a deprecated idea and should not be implemented in new connectors. Fluent flow control and smartNotifierHandler should be used instead.

req.hull.enqueue("jobName", { user: [] }, options = {});

A function added to context by Queue Module. It allows to perform tasks in an async manner. The queue is processed in background in a sequential way, it allows to:

respond quickly in the express application actions (they just queue the work)
split the workload into smaller chunks (e.g. for extract parsing)
control the concurrency - most of the SERVICE APIs have rate limits

Full API reference

metric

An object added to context by Instrumentation Module. It allows to send data to metrics service. It's being initiated in the right context, and expose following methods:

req.hull.metric.value("metricName", metricValue = 1);
req.hull.metric.increment("metricName", incrementValue = 1); // increments the metric value
req.hull.metric.event("eventName", { text = "", properties = {} });

An object added to context by the Instrumentation Module. It allows to send data to the metrics service. Full API reference

helpers

Helpers are just a set of simple functions added to the Context Object which make common operation easier to perform. They all follow context management convention but the functions can be also used in a standalone manner:

const { updateSettings } = require("hull/lib/helpers");

app.post("/request", (req, res) => {
  updateSettings(req.hull, { called: true });
  // or:
  req.hull.helpers.updateSettings({ called: true });
});

Beside of connector setting updating, they also simplify working with outgoing extracts.

All helpers are listed in API REFERENCE

service

A namespace reserved for connector developer to inject a custom modules. When the connector base code evolves, the best technique to keep it maintainable is to split it into a set of functions or classes. The service namespace is reserved for the purpose and should be used together with use method on connector instance to apply custom middleware. That should be an object with custom structure adjusted to specific connector needs and scale:

// custom middleware creating the `service` param
connector.use((req, res, next) => {
  req.hull.service = {
    customFunction: customFunction.bind(req.hull),
    customModule: new CustomClass(req.hull)
  };
  next();
});

connector.setupApp(app);

app.get("/action", (req, res) => {
  const { service } = req.hull;
  service.customFunction(req.query.user_id);
  // or
  service.customModule.customMethod(req.query.user_id);
});

We strongly advice to follow our context management convention which make it easy to keep functions and classes signatures clean and standard.

message

Optional - set if there is a sns message incoming.

It contains the raw, message object - should not be used directly by the connector, req.hull.notification is added for that purpose.

Type: "Notification",
Subject: "user_report:update",
Message: "{\"user\":{}}"

notification

Optional - if the incoming message type if Notification, then the messaged is parsed and set to notification.

subject: "user_report:update",
timestamp: new Date(message.Timestamp),
paload: { user: {} }

smartNotifierResponse

Use setFlowControl to instruct the Smart notifier how to handle backpressure.

ctx.smartNotifierResponse.setFlowControl({
  type: 'next',
  size: 100,
  in: 5000
});

Configuration resolve strategy

During Context Object building important step is how Hull Client configuration is read. The whole strategy is descibed below step-by-step.

Here is what happens when your Express app receives a query:

If a config object is found in req.hull.config steps 2 and 3 are skipped.
If a token is present in req.hull.token, the middleware will try to use the hostSecret to decrypt it and set req.hull.config.
If the query string (req.query) contains id, secret, organization, they will be stored in req.hull.config.
After this, if a valid configuration is available in req.hull.config, a Hull Client instance will be created and stored in req.hull.client.
When this is done, then the Ship will be fetched and stored in req.hull.ship

If there is a req.hull.cache registered in the Request Context Object, it will be used to cache the ship object. For more details see Context Object Documentation

If the configuration or the secret is invalid, an error will be thrown that you can catch using express error handlers.

Custom middleware

The Hull.Connector architecture gives a developer 3 places to inject custom middleware:

At the very beginning of the middleware stack - just after const app = express(); - this is a good place to initialy modify the incoming request, e.g. set the req.hull.token from custom property
After the Context Object is built - after calling setupApp(app) - all context object would be initiated, but req.hull.client, req.hull.segments and req.hull.ship will be present only if credentials are passed.
Before the closing startApp(app) call which internally calls app.listen()

Context management convention

The context object is treated by the Hull.Connector as a dependency injection container which carries on all required dependencies to be used in actions, jobs or custom methods.

This library sticks to a the following convention of managing the context object:

Functions

All functions take context as a first argument:

function getProperties(context, prop) {
  cons { client } = context;
  return client.get("/properties", { prop });
}

This allow binding such functions to the context and using bound version

const getProp = getProperties.bind(null, context);
getProp("test") === getProperties(context, "test");

Classes

In case of a class the context is the one and only argument:

class ServiceAgent {
  constructor(context) {
    this.client = context.client;
  }
}

All functions and classes listed in API reference and available in the context object follow this convention when used from contex object they will be already bound, so you don't need to provide the first argument when using them.

Incoming data flow

To get data into platform we need to use traits or track methods from HullClient (see details here). When using Hull.Connector we have the client initialized in the correct context so we can use it right away in side any HTTP request handler.

Let's write the simplest possible HTTP endpoint on the connector to fetch some users:

const app = express();
const connector = new Hull.Connector();

connector.setupApp(app);

app.get("/fetch-users", (req, res) => {
  const ctx = req.hull;
  const { api_key } = ctx.ship.private_settings;

  // let's try to get some data from 3rd party API
  const customApiClient = new CustomApiClient(api_key);

  customApiClient.fetchUsers()
    .then(users => {
      return users.map((user) => {
        return ctx.client.asUser(user.ident).traits(user.attributes);
      });
    })
    .then(() => {
      res.end("ok");
    });
});

connector.startApp(app);

Then we can create a button on the connector dashboard to run it or call it from any other place. The only requirement is that the enpoint is called with credentials according to the configuration resolve strategy.

Schedules

If you want to run specific endpoint with a selected interval you can use schedules param of the manifest.json:

{
 "schedules": [
    {
      "url": "/fetch-users",
      "type": "cron",
      "value": "*/5 * * * *"
    }
  ]
}

This way selected connector endpoint would be run at every 5th minute.

Outgoing data flow

To peform operations on in response to new data coming in or being updated on Hull Platform we use two means of communications - notifications which are triggered on per user/event/change basis or batch extracts which can be sent manually from the dashboard UI or requested by the connector.

Notifications

All events triggered on user base result in a notification hitting specified connector endpoint. Current Hull Connector version supports two generations of those notifications - legacy and new "smart-notifier". Following guide assume you are using the new generation.

To subscribe to platform notifications, define the endpoint in connector manifest.json:

{
  "tags": ["smart-notifier"],
  "subscriptions": [{
    "url": "/smart-notifier"
  }]
}

Then in ExpressJS server definition we need to pick smartNotifierHandler from utils directory:

const { smartNotifierHandler } = require("hull/lib/utils");

const app = express();
const connector = new Hull.Connector();

connector.setupApp(app);

app.use("/smart-notifier", smartNotifierHandler({
  handlers: {
    "user:update": (ctx, messages = []) => {
      // more about `smartNotifierResponse` below
      ctx.smartNotifierResponse.setFlowControl({
        type: "next",
        size: 100,
        in: 5000
      });
      return Promise.resolve();
    }
  },
  options: {
    groupTraits: false
  }
}));

connector.startApp(app);

The user:update handler will be run with batches of notification messages coming from platform. User update message is a json object which is grouping together all events and changes which happened on the specic user since the previous notification. The structure of the single message is defined in this Flow Type.

Inside the handler you can use any object from the Context Object. Remember that the handler needs to return a valid promise at the end of it's operations.

Full information on smartNotifierHandler is available in API REFERENCE.

FlowControl

Smart-notifier generation of notifications delivery allows us to setup flow control which define pace at which connector will be called with new messages:

ctx.smartNotifierResponse.setFlowControl({
  type: "next", // `next` or `retry`, defines next flow action
  size: 1000, // only for `next` - number of messages for next notification
  in: 1000, // delay for next flow step in ms
  at: 1501062782 // time to trigger next flow step
});

FlowControl is an element of the SmartNotifierResponse. When the HTTP response is built it has the following structure

// response body:
{
  flow_control: {
    type: "next",
    in: 1000
  },
  metrics: []
}

The Defaults are the following:

// for a resolved, successful promise:
{
  type: "next",
  size: 1,
  in: 1000
}

// for a rejected, erroneous promise:
{
  type: "retry",
  in: 1000
}

Batch extracts

Second way of operating on Hull user base it to process batch extracts.

In addition to event notifications Hull supports sending extracts of users and accounts. These extracts can be triggered via manual user action on the dashboard or can be programmatically requested from the Connector logic (see requestExtract helper). The Connector will expose the manual batches action if your manifest.json includes a batch or batch-accounts tag :

{
  "tags" : [ "batch", "batch-accounts" ]
}

In both cases the batch extracts are processed by the user:update and account:update handlers. The extract is split into smaller chunks using options.maxSize. Only traits and segments are exposed in the extracted lines, events and changes are never sent.

In addition, to let the handler function detect whether it is processing a batch extract or notifications, a third argument is passed- in case of notifications it is undefined, otherwise it includes query and body parameters from req object.

Notification of batches, when the extracts are ready are sent as a POST request on the /batch and /batch-accounts endpoints respectively.

app.post("/batch", notifHandler({
  options: {
    maxSize: 100,
    groupTraits: false
  },
  handers: {
    "user:update": ({ hull }, users) => {
      hull.logger.info("Get users", users);
    }
  }
}));

Connector status

Platform API comes with an endpoint where connector can post it's custom checks performed on settings and/or 3rd party api. The resulted should be posted to an endpoint but for testing and debugging purposes we also respond with the results.

Here comes example status implementation:

app.all("/status", (req, res) => {
  const { ship, client } = req.hull;
  const messages = [];
  let status = "ok";

  const requiredSetting = _.get(ship.private_settings, "required_setting");
  if (code === undefined) {
    status = "warning";
    messages.push("Required setting is not set.");
  }

  res.json({ messages, status });
  return client.put(`${req.hull.ship.id}/status`, { status, messages });
});

Then to make it being run in background we can use a schedule entry in manifest.json:

{
  "schedules": [
    {
      "url": "/status",
      "type": "cron",
      "value": "*/30 * * * *"
    }
  ]
}

Installation & Authorization

First step of connector installation is done automatically by the platform and the only needed part from connector end is manifest.json file.

However typically after the installation we want that the connector is authorized with the 3rd party API.

Hull Node comes with packaged authentication handler using Passport - the utility is called oAuthHandler and you can find documentation here.

Utilities

Beside of Hull.Connector class and Context Object all other public API elements of this library is exposed as Utils which are standalone functions to be picked one-by-one and used in custom connector code.

List of all utilities are available here

Superagent plugins

Hull Node promotes using SuperAgent as a core HTTP client. We provide two plugins to add more instrumentation over the requests.

superagentErrorPlugin
superagentInstrumentationPlugin
superagentUrlTemplatePlugin

Infrastructure

The connector internally uses infrastructure modules to support its operation on application process level and provide some of the Context Object elements like cache, metric and enqueue. See following API REFERENCE docs to see what is the default behavior and how to change it:

Instrumentation (for gathering metrics)
Cache (for caching ship object, segment lists and custom elements)
Queue (for internal queueing purposes)
Batcher (for internal incoming traffing grouping)

Worker

More complex connectors usually need a background worker to split its operation into smaller tasks to spread the workload.

This is generally a deprecated idea and should not be implemented in new connectors. Fluent flow control and smartNotifierHandler should be used instead.

const express = require("express");
const Hull = require("hull");

const app = express();

const connector = new Hull.Connector({ hostSecret });
// apply connector related features to the application
connector.setupApp(app);

connector.worker({
  customJob: (ctx, payload) => {
    // process payload.users
  }
});
app.post("/fetch-all", (req, res) => {
  req.hull.enqueue('customJob', { users: [] });
});
connector.startApp(app, port);
connector.startWorker((queueName = 'queueApp'));

Flow annotations

When using a flow enabled project, we recommend using flow types provided by hull-node. You can import them in your source files directly from hull module and use import type flow structure:

// @flow
import type { THullObject } from "hull";

parseHullObject(user: THullObject) {
  // ...
}

See src/lib/types directory for a full list of available types.

Error handling

All handlers for outgoing traffic are expecting to return a promise. Resolution or rejection of the promise triggers different behavior of error handling. Default JS errors are treated as unhandled errors, the same applies for any unhandled exceptions thrown from the handler code.

Hull Connector provides two other error classes TransientError and LogicError which are handled by internals of the SDK in a different way.

The convention is to filter known issues and categorize them into transient or logic errors categories. All unknown errors will default to unhandled errors.

Unhandled error

Default, native Javascript error.

context	behavior
smart-notifier response	retry
other endpoints response	error
status code	500
sentry exception report	yes
datadog metrics	no

app.use("/smart-notifier-handler", smartNotifierHandler({
  handlers: {
    "user:update": (ctx, messages) => {
      return Promise.reject(new Error("Error message"));
    }
  }
}));

Transient error

This is an error which is known to connector developer. It's an error which is transient and request retry should be able to overcome the issue. It comes with 3 subclasses to mark specifc scenarios which are related to time when the error should be resolved.

RateLimitError
ConfigurationError
RecoverableError

The retry strategy is currently the same as for unhandled error, but it's handled better in terms of monitoring.

context	behavior
smart-notifier response	retry
other endpoints response	error
status code	400
sentry exception report	no
datadog metrics	yes

const { TransientError } = require("hull/lib/errors");

app.use("/smart-notifier-handler", smartNotifierHandler({
  handlers: {
    "user:update": (ctx, messages) => {
      ctx.smartNotifierResponse.setFlowControl({

      });

      return Promise.reject(new TransientError("Error message"));
    }
  }
}));

Logic error

This is an error which needs to be handled by connector implementation and as a result the returned promised must not be rejected.

IMPORTANT: Rejecting or throwing this error without try/catch block will be treated as unhandled error.

context	behavior
smart-notifier response	next
other endpoints response	success
status code	200
sentry exception report	no
datadog metrics	no

const { LogicError } = require("hull/lib/errors");

app.use("/smart-notifier-handler", smartNotifierHandler({
  handlers: {
    "user:update": (ctx, messages) => {
      return (() => {
        return Promise.reject(new LogicError("Validation error"));
      })
      .catch((err) => {
        if (err.name === "LogicError") {
          // log outgoing.user.error
          return Promise.resolve();
        }
        return Promise.reject(err);
      });
    }
  }
}));

Files

README.md

Latest commit

History

README.md

File metadata and controls

Overview

Hull Client

Hull Middleware

Hull Connector

Hull.Middleware

Hull.Connector

Initialization

Setup Helpers

Context Object

Base Context - set by Hull.Middleware

requestId

config

token

client

ship

hostname

options

Extended Context - set by Hull.Connector

connectorConfig

segments

cache

enqueue

metric

helpers

service

message

notification

smartNotifierResponse

Configuration resolve strategy

Custom middleware

Context management convention

Functions

Classes

Incoming data flow

Schedules

Outgoing data flow

Notifications

FlowControl

Batch extracts

Connector status

Installation & Authorization

Utilities

Superagent plugins

Infrastructure

Worker

Flow annotations

Error handling

Unhandled error

Transient error

Logic error

Extended Context - set by `Hull.Connector`