Truedat Quality Service

TdQx is a back-end service that supports the third version of Quality used by Truedat.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

Install dependencies with mix deps.get

To start your Phoenix server:

Installing

• Create and migrate your database with mix ecto.setup
• Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server
• Now you can visit localhost:4010 from your browser.

Running the tests

Run all aplication tests with mix test

Deployment

Ready to run in production? Please check our deployment guides.

Environment variables

• REDIS_AUDIT_STREAM_MAXLEN (Optional) Maximum length for Redis audit stream. Default: 100
• REDIS_STREAM_MAXLEN (Optional) Maximum length for Redis stream. Default: 100

SSL conection

• DB_SSL: boolean value, to enable TSL config, by default is false.
• DB_SSL_CACERTFILE: path of the certification authority cert file "/path/to/ca.crt".
• DB_SSL_VERSION: available versions are tlsv1.2, tlsv1.3 by default is tlsv1.2.
• DB_SSL_CLIENT_CERT: Path to the client SSL certificate file.
• DB_SSL_CLIENT_KEY: Path to the client SSL private key file.
• DB_SSL_VERIFY: This option specifies whether certificates are to be verified.

Elastic bulk page size configuration

-BULK_PAGE_SIZE_QUALITY_CONTROLS: default 5000 -BULK_PAGE_SIZE_SCORE_GROUPS: default 5000 -BULK_PAGE_SIZE_QUALITY_CONTROL_VERSIONS: default 5000

Elastic aggregations

• The aggregation variables are defined as follows: AGG_<AGGREGATION_NAME>_SIZE

Built With

• Phoenix - Web framework
• Ecto - Phoenix and Ecto integration
• Postgrex - PostgreSQL driver for Elixir
• Cowboy - HTTP server for Erlang/OTP
• Credo - Static code analysis tool for the Elixir language
• cors_plug - Plug for CORS support
• ex_machina - A factory library for test data

Environment variables

Elastic bulk page size configuration

• BULK_PAGE_SIZE_QUALITY_CONTROL_VERSIONS

ElasticSearch authentication

(Optional) Basic HTTP authentication

These environment variables will add the Authentication header on each request with value Basic <ES_USERNAME>:<ES_PASSWORD>

• ES_USERNAME: Username
• ES_PASSWORD: Password

(Optional) ApiKey authentication

This environment variables will add the Authentication header on each request with value ApiKey <ES_API_KEY>

• ES_API_KEY: ApiKey

(Optional) HTTP SSL Configuration (Normally required for ApiKey authentication)

These environment variables will configure CA Certificates for HTTPS requests

• ES_SSL: [true | false] required to activate following options
• ES_SSL_CACERTFILE: (Optional) Indicate the cacert file path. If not set, a certfile will be automatically generated by :certifi.cacertfile()
• ES_SSL_VERIFY: (Optional) [verify_peer | verify_none] defaults to verify_none

ElasticSearch Force Merge Configuration

These environment variables control the force merge operation for ElasticSearch indices, which optimizes index performance by merging segments.

• ES_WAIT_FOR_COMPLETION:

  • Purpose: Controls whether the force merge operation should wait for completion before returning
  • Default: nil (no wait)
  • Usage: When set to true, the operation will wait until the force merge is complete before returning. When false or nil, the operation returns immediately and runs asynchronously
  • Performance: Setting to true ensures the operation is complete but may cause longer response times

• ES_MAX_NUM_SEGMENTS:

  • Purpose: Specifies the maximum number of segments to merge down to
  • Default: 5
  • Usage: Controls how aggressively the force merge operation consolidates segments. Lower values result in fewer, larger segments
  • Performance: Fewer segments generally improve search performance but may increase memory usage during the merge operation

Authors

• Bluetab Solutions Group, SL - Initial work - Bluetab

See also the list of contributors who participated in this project.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

In order to use this software, it is necessary that, depending on the type of functionality that you want to obtain, it is assembled with other software whose license may be governed by other terms different than the GNU General Public License version 3 or later. In that case, it will be absolutely necessary that, in order to make a correct use of the software to be assembled, you give compliance with the rules of the concrete license (of Free Software or Open Source Software) of use in each case, as well as, where appropriate, obtaining of the permits that are necessary for these appropriate purposes.