SQS Upload

SQS Data Ingestion

The mParticle platform can receive large volumes of data sent to an Amazon SQS queue - use this page to learn how to do so with the objects provided in the mParticle Node SDK.

1. AWS SDK Setup

To post messages to an SQS queue, you will need to be familiar with the AWS SDK for Node. You can read more about the aws-sdk here, or jump to the code samples below.

2. AWS Authentication

To support cross-account access, you must set up an IAM role and a user/credentials with the permission to assume that role. mParticle will then grant access to the role.

1. Create an IAM role

2. Edit or create a new policy for a user to assume the role

3. The AWS SDK can automatically assume the role you created. You can configure the profile to use as in this example:

   // In config.json
   { "accessKeyId": <YOUR_ACCESS_KEY_ID>, "secretAccessKey": <YOUR_SECRET_ACCESS_KEY>, "region": "us-east-1" }

   See more about how to configure your access keys here

4. Send the full ARN of the role to mParticle, and we will provide you with an SQS URL to use.

3. Constructing a Batch

SQS data ingestion allows you to send lists of Batch objects, or single Batch objects, following the same schema as our HTTP server-to-server API. Use the models provided by the Node SDK to construct your Batch objects before serializing them for transmission to AWS.

Set the api_key property of each Batch to assign it to an mParticle Input, as well as the environment property:

var batch = new mParticle.Batch();
batch.api_key = 'REPLACE ME';
batch.environment = 'development';

Contact to mParticle's implementation team and consult your mParticle data plan to determine which events, user attributes, device or user identities, and other Batch properties to include in your upload.

Similar to the mParticle HTTP server-to-server API, you may send a single Batch object or a list of Batch objects in a single SQS message.

4. Serialize and Transmit to SQS

Serialize the Batch objects into a JSON message

// Include multiple batches per upload 
var batches = [batch];

// or just send a single batch
// var batches = batch;

// Serialize the upload - SQS allows for up to 256KB per message
var message = JSON.stringify(batches);

Retrieve a reference to the queue

Using the profile you set up in ./config.json above, grab a reference to the mParticle queue:

var AWS = require('aws-sdk');
AWS.config.loadFromPath('./config.json');
var sqs = new AWS.SQS();

// PLEASE FIX (you provide the queue url in the params object)
// mParticle to provide the queue name
// queue = sqs.get_queue_by_name(QueueName='REPLACE ME')

var params = {
     DelaySeconds: 10, // Optional
     QueueUrl: "SQS_QUEUE_URL"
};

Upload the message

mParticle requires the following SQS message attributes to be included with each message:

Key	Value	Data Type
message-type	s2sbatch	String
message-format	json/text	String
message-version	2	String

params.MessageBody = message;

// these attributes are required
params.MessageAttributes = {
    'message-type':{
        'StringValue':'s2sbatch',
        'DataType':'String'
    },
    'message-format':{
        'StringValue':'json/text',
        'DataType':'String'
    },
    'message-version':{
        'StringValue':'2',
        'DataType':'String'
    }
};

sqs.sendMessage(params, function (err, data) {
    if (err) {
        console.log("Error", err);
    } else {
        console.log("Success", data.MessageId);
    }
});

Full Code Sample

var AWS = require('aws-sdk');
var mParticle = require('mparticle');

var batch = new mParticle.Batch();
batch.api_key = 'REPLACE ME';
batch.environment = 'development';

// construct a batch object per user
var appEvent = new mParticle.AppEvent(mParticle.AppEvent.CustomEventType.navigation, 'navigation');
batch.events = [new mParticle.SessionStartEvent(), appEvent, new mParticle.SessionEndEvent()];
batch.user_attributes = {'eyes':'brown','favorite_sports':['football','tennis']} ;

// Include multiple batches per upload 
var batches = [batch];

// or just send a single batch
// var batches = batch;

// Serialize the upload - SQS allows for up to 256KB per message
AWS.config.loadFromPath('./config.json');
var sqs = new AWS.SQS();

var message = JSON.stringify(batches);

var params = {
     DelaySeconds: 10, // Optional
     QueueUrl: "SQS_QUEUE_URL"
};

params.MessageBody = message;

// these attributes are required
params.MessageAttributes = {
    'message-type':{
        'StringValue':'s2sbatch',
        'DataType':'String'
    },
    'message-format':{
        'StringValue':'json/text',
        'DataType':'String'
    },
    'message-version':{
        'StringValue':'2',
        'DataType':'String'
    }
};

sqs.sendMessage(params, function (err, data) {
    if (err) {
        console.log("Error", err);
    } else {
        console.log("Success", data.MessageId);
    }
});