Ingest parsers for Elasticsearch which cater for esoteric log formats in the drilling industry
eLAStify_s3.py
scrapes an S3 bucket for LAS files then downloads and parses them before indexing them into Elasticsearch.
./eLAStify_s3.py --help
usage: eLAStify.py [-h] --es-url elasticsearch.mydomain.com [--es-port 9200]
[--index las_data] [--insecure False] [--user elastic]
[--password Pa55w0rd] [--ca-cert /path/to/ca/ca.crt]
--bucket my-bucket
Processes LAS data files and indexes them into an Elasticsearch cluster.
optional arguments:
-h, --help show this help message and exit
--es-url elasticsearch.mydomain.com
the Elasticsearch cluster URL
--es-port 9200 the port Elasticsearch listens on
--index las_data the Elasticsearch cluster URL
--insecure False set to True (not lowercase true) if connecting to a
cluster not secured with SSL/TLS
--user elastic not needed if connecting to an unsecured cluster
--password Pa55w0rd not needed if connecting to an unsecured cluster
--ca-cert /path/to/ca/ca.crt
not needed if connecting to a cluster not secured with
SSL/TLS
--bucket my-bucket name of the S3 bucket to retrieve files from
top2es_s3.py
performs the same function but on Wellbore / TOPS data format files.
./top2es_s3.py --help
usage: top2es.py [-h] --es-url elasticsearch.mydomain.com [--es-port 9200]
[--index tops_data] [--insecure False] [--user elastic]
[--password Pa55w0rd] [--ca-cert /path/to/ca/ca.crt] --bucket
my-bucket
Processes Wellbore / TOPS data files and indexes them into an Elasticsearch
cluster.
optional arguments:
-h, --help show this help message and exit
--es-url elasticsearch.mydomain.com
the Elasticsearch cluster URL
--es-port 9200 the port Elasticsearch listens on
--index tops_data the index the documents should be written to
--insecure False set to True (not lowercase true) if connecting to a
cluster not secured with SSL/TLS
--user elastic not needed if connecting to an unsecured cluster
--password Pa55w0rd not needed if connecting to an unsecured cluster
--ca-cert /path/to/ca/ca.crt
not needed if connecting to a cluster not secured with
SSL/TLS
--bucket my-bucket name of the S3 bucket to retrieve files from
the _localfile
variants perform the same functions on a local directory; for these replace the --bucket
flag with --directory
and pass the desired location to search (NB: search is recursive so subfolders will also be checked for matching files).