Query JSON and NDJSON files on Amazon S3
The selectObjectContent API allows to easily query JSON and NDJSON data from S3. This node module provides a wrapper for this method, exposing the data as an aggregated result as a Promise.
See the official AWS documentation for more information.
$ npm install s3-query-json
const {query} = require('s3-query-json');
query('foobarbaz', 'users.ndjson', 'SELECT s.name FROM S3Object s')
.then(data => console.log(data) // [{"name": "unicorn"}, {"name": "rainbow"}])
query('foobarbaz', 'users.ndjson', 'SELECT s.name FROM S3Object s', {documentType: 'NDJSON', compressionType: 'GZIP', stream: false, delimiter: '\n', scanRange: {start: 1, end: 100}})
.then(data => console.log(data) // [{"name": "unicorn"}, {"name": "rainbow"}])Type: String
The name of the bucket containing your object.
Type: String
The key of the S3 object, either JSON or NDJSON format.
Type: String
SQL query to be executed. Read the documentation on which SQL operations are allowed.
Type: Object
Configuration object.
Type: String
Default: NDJSON
Accepts either JSON or NDJSON as input.
Type: String
Default: NONE
Compression used in the S3 object. Either NONE, GZIP or BZIP2
Type: Boolean
Default: false
Flag to indicate if the function returns a Promise with a original readstream. This allows more advanced custom processing.
Type: string
Default: \n
Delimiter with a maximum length of 1 to separate the records that are being returned. To avoid difficult parsing, use a delimiter with no collision with the records returned.
Type: Object
Default: undefined
This property can only be used with non-compressed (ND)JSON files. Allows to scan a specified range of bytes. See the documentation for more information. When scanRange is provided, atleast one of keys start or end must be provided.
Type: Number
Inclusive start of the range. Starts from 0
Type: Number
Inclusive end of the range. This is allowed to be out of bound of the object size.
MIT © Simon Jang