db-extractor

Code quality analysis and Build Status

What is this repository for?

Extract information from databases (MySQL, MariaDB, SAP HANA to start with, other will be implemented later) using a combination of:

• extraction sequences file (JSON format) that is easy enough to create and maintain but also provide very complex features to be set;
• source system file (JSON format) to keep a central list of servers and/or databases to connect to that can be shared between people;
• user settings file (JSON format) to keep a central list of credentials that is not to be shared with anyone or maybe with a small group of people;

Features implemented

• Ability to extract from a single source system or multiple using 1 JSON extraction sequence file;
• Ability to extract a single or multiple query for each source system using same JSON extraction sequence file;
• Ability to extract a single or multiple files using sessions for each query where parameters can be specified (currently on CSV and Excel file format are supported, other will follow);
• Multi-language (English, Italian, Romanian);
• Enhance behaviour choices so that besides existing 'skip-if-output-file-exists' and 'overwrite-if-output-file-exists' to have the option to specify to overwrite but only if the file is older than any choice of a CalculatedDate expression is given, as this is very useful when extracting large amount of data over VPN in small pieces and VPN drops (could mean already extracted pieces would be already skipped as not older than threshold imposed);

Supported File Types/Formats

• Comma Separated Values (with ability to specify a custom separator of your preference)
• Excel 2013+
• JSON
• Parquet (with compression algorithms: brotli, gzip, snappy and none to choose from)
• Pickle (with compression algorithms: bz2, gzip, xz, zip and none to choose from with as special value as "infer" to detect automatically the correct one from provided file extension)

Who do I talk to?

Repository owner is: Daniel Popiniuc

Installation

Installation can be completed in few steps as follows:

• Ensure you have git available to your system:

    $ git --version

If you get an error depending on your system you need to install it.

For Windows you can do so from Git for Windows;

• Download this project from Github:

    $ git clone https://github.com/danielgp/db-extractor <local_path_of_this_package>

conventions used:

<content_within_html_tags> = variables to be replaced with user values relevant strings

• Create a Python Virtual Environment using following command executed from project root folder:

    $ python(.exe) -m venv <local_folder_on_your_computer_for_this_package>/virtual_environment/

• Upgrade pip (PIP is a package manager for Python packages) using following command executed from newly created virtual environment and Scripts sub-folder:

    $ <local_path_of_this_package>/virtual_environment/Scripts/python(.exe) -m pip install --upgrade pip

• Install project prerequisites using following command executed from project root folder:

    $ <local_path_of_this_package>pip install -r requirements.txt

• Ensure all localization source files are compile properly in order for the package to work properly

    $ <local_path_of_this_package>/virtual_environment/Scripts/python(.exe) <local_path_of_this_package>/sources/project_locale/localizations_compile.py

Maintaining local package up-to-date

Once the package is installed is quite important to keep up with latest releases as such are addressing important code improvements and potential security issues, and this can be achieved by following command:

    $ git --work-tree=<local_path_of_this_package> --git-dir=<local_path_of_this_package>/.git/ --no-pager pull origin master

• conventions used:
  • <content_within_html_tags> = variables to be replaced with user values relevant strings

Usage

    $ python <local_path_of_this_package>/sources/extractor.py --input-source-system-file <input_source_system_file_name> --input-credentials-file <input_credentials_file_name> --input-extracting-sequence-file <input_extracting_sequence_file_name> (--output-log-file <full_path_and_file_name_to_log_running_details>)

conventions used:

(content_within_round_parenthesis) = optional <content_within_html_tags> = variables to be replaced with user values relevant strings single vertical pipeline = separator for alternative options

Example of usage

    $ python sources/extractor.py --input-source-system-file samples/sample---server-config.json --input-credentials-file samples/sample---user-settings.json --input-extracting-sequence-file samples/sample---list-of-fields.json --output-log-file samples/sample---list-of-fields.log

Code of conduct

Use CODE_OF_CONDUCT.md

Features already raised

• Implement ability to store extracted result-set into HTML format file;

Features to request template

Use feature_request.md

Bug report template

Use bug_report.md

Required software/drivers/configurations

see readme_software.md

Used references

see readme_reference.md