Skip to content

Library for identification of items, events or observations which do not conform to an expected pattern or other items in a dataset

Notifications You must be signed in to change notification settings

rreinold/anomaly-detection-template

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

46 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ipm package: anomaly-detection

Overview

Run a self-training anomaly detection algorithm against a given dataset. The selected algorithm is Moving Median Decomposition as seen in this whitepaper for Anomaly Detection for Predictive Maintenance. This Library is for identification of items, events or observations which do not conform to an expected pattern or other items in a dataset.

This is an ipm package, which contains one or more reusable assets within the ipm Community. The 'package.json' in this repo is a ipm spec's package.json, here, which is a superset of npm's package.json spec, here.

Browse ipm Packages

Setup

Basic Usage (No Alerts)

Ready out-of-the-box

Advanced Usage (With Alerts)

Requires API keys from Email and SMS Alerts via SendGrid and Twilio.

See AnomalyDetectionConstants Library.

Usage

Portals

AnomalyDetection - Portal for viewing and configuring live datafeeds, rules, and alerts

Code Services

ExampleDetectAnomaly - Runs anomaly detection against an example dataset, returns an array of Anomalies with metadata.

Code Libraries

AnomalyDetection - Core library for detecting anomalies in datasets AnomalyDetectionConstants - Optional credentials for Alerting (SMS via Twilio, Email via Sendgrid) jStat - Dependency on jstat-statistics-toolkit TwilioSMS - Dependency on twilio-sms-library SendGridEmail - Dependency on(https://github.com/rreinold/clearblade-sendgrid-email-integration)

Collections

Anomalies - Tracks all detected anomalies AnomalyConfiguration - Stores user-defined rules and alerts for Anomaly Detection

API

Members

Functions

AnomalyDetection()

Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf

Run a self-training anomaly detection algorithm against a given dataset

Twilio(user, pass, sourceNumber)

Sends a text message using Twilio's REST API.

DetectAnomaly()AnomalyVisual

Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}

ExampleDetectAnomaly()Array.<Anomaly>

Example logic for detecting anomalies in a dataset of sensor values

SaveAnomalyConfiguration()

Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection

SendAlert()

This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection

  • Fetches the anomaly row
  • Checks for alerting configuration
  • Sends email, text alert
SimulateSensorFeed()

Simulates a sensor feed from a sensor network of temperature sensors Publishes NUM_MESSAGES messages to TOPIC

Typedefs

Calibration
Anomaly

SEND_GRID_API_URI

Creator: Robert Reinold Updated: 2017-10-02T00:00:00Z Version: v2.0 Tags: email, sendgrid, marketing, mail, api, rest, http

Usage:

  1. Create a free SendGrid Account.
  2. Log into your SendGrid account, and view the Settings > API Keys tab. Create an API Key with full access to "Mail Send" rights.
  3. Replace <SEND_GRID_API_KEY> with SendGrid API key
  4. Replace <ORIGIN_EMAIL_ADDRESS> with your desired email address. This will be the 'sender' of the email.
  5. Add 'SendGridEmail' as a dependency to your code services (Settings > Requires > Add)

Kind: global variable

AnomalyDetection()

Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf

Run a self-training anomaly detection algorithm against a given dataset

Kind: global function
Parameter: number[] dataset array of numbers

AnomalyDetection~calibrate()

Use precomputed anomaly detection calibration profile This can be used to speed up performance for real-time anomaly detection

Kind: inner method of AnomalyDetection

AnomalyDetection~detect() ⇒ Array.<Anomaly>

Run the algorithm against the provided dataset.

Note: This is the most computation-heavy method in this library

Kind: inner method of AnomalyDetection
Returns: Array.<Anomaly> - anomalies - the list of anomalies found in dataset
Parameter: number strictnessOverride (optional) Set the strictness of the anomaly detection

AnomalyDetection~getCalibration() ⇒ Calibration

Fetch the computed anomaly detection calibration parameters

Kind: inner method of AnomalyDetection
Returns: Calibration - calibration

DetectAnomaly() ⇒ AnomalyVisual

Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}

Kind: global function
Returns: AnomalyVisual - data to visualize this anomaly detection

ExampleDetectAnomaly() ⇒ Array.<Anomaly>

Example logic for detecting anomalies in a dataset of sensor values

Kind: global function
Returns: Array.<Anomaly> - anomalies

SaveAnomalyConfiguration()

Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection

Kind: global function
Parameter: AnomalyConfiguration anomalyConfiguration - map of settings to update

SendAlert()

This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection

  • Fetches the anomaly row
  • Checks for alerting configuration
  • Sends email, text alert

Kind: global function

SimulateSensorFeed()

Simulates a sensor feed from a sensor network of temperature sensors Publishes NUM_MESSAGES messages to TOPIC

Kind: global function

Calibration

Kind: global typedef
Properties

Name Type Description
min number min value within threshold
max number max value within threshold
strictness number how strict the threshold is against the dataset
medians number intermediary dataset representing the moving medians
pointsPerWindow number calibration, number of points per processing window. may be increased for larger datasets
numWindows number calibration metric derived from pointsPerWindow

Anomaly

Kind: global typedef
Properties

Name Type Description
index number index in the provided dataset
value number value of the data point that is detected as an anomaly

About

Library for identification of items, events or observations which do not conform to an expected pattern or other items in a dataset

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published