NEWS

# Releases

  ## grimoire-elk 0.109.4 - (2023-12-19)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.109.3 - (2023-11-28)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.109.2 - (2023-11-17)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.109.1 - (2023-11-03)
  
  * Update Poetry's package dependencies

## grimoire-elk 0.109.0 - (2023-10-20)

**New features:**

 * Anonymize repository URLs in graal backend\
   The raw data generated by the graal backend had non-anonymized URLs on
   the meta fields 'origin' and 'tag'. This was necessary to generate
   enriched data for non-public repositories.
 * Bug description in Bugzilla\
   Include the description of the bugs in Bugzilla. The new fields are
   'description' and 'description_analyzed'. The second allows to query
   any of the words of the description.
 * SortingHat backend name per data source\
   Allow to define the name of the backend for the identities stored in
   SortingHat. From now on github2 is renamed as github in SortingHat
   identities.

**Bug fixes:**

 * Individual `id` incorrectly assigned\
   Correct a bug that results in enriched items using the individual ID
   as the identity ID, and vice versa.
 * Error in OpenSearch response stops demography study\
   Fix an issue that stops demography study when OpenSearch occasionally
   provides a response without some expected fields.
 * Undefined identities in GitHub comments\
   Fix a bug that causes certain identities from commentaries to not be
   imported into SortingHat, resulting in them appearing as UNDEFINED in
   OpenSearch.

**Dependencies updateds:**

 * Add Python 3.9 and 3.10 and drop 3.7 support\
   Python 3.7 reached the end of life phase on June 27 2023 and is no
   longer supported.
 * Bump urllib3 version\
   Bump urllib3 from ^1.26 to ^2.0.7
 * Remove Python pymsql dependency\
   Remove the dependency with PyMySQL because it is not used anymore.


  ## grimoire-elk 0.108.1 - (2023-08-06)
  
  * Update Poetry's package dependencies

## grimoire-elk 0.108.0 - (2023-07-23)

**Bug fixes:**

 * Enrich items with multiple enrollments\
   The `date` field must be a string since improved performance by
   decreasing calls to the `str_to_datetime` method.

**Performance improvements:**

 * Load identities performance\
   Improve the load identities process performance by reducing the number
   of queries to SortingHat.


## grimoire-elk 0.107.0 - (2023-07-11)

**Performance improvements:**

 * Enrichment processing time reduced by 50%\
   The general performance was improved reducing the number of calls to
   the identities manager (i.g. `SortingHat`). There were some deprecated
   calls that weren't needed any longer and also, we increased the cache
   of individuals in ELKs. We were also able to reduce the processing
   time of the Git backend by converting commit dates only once.


## grimoire-elk 0.106.0 - (2023-06-28)

**Performance improvements:**

 * Refreshing identities performance\
   Decrease the time required to execute the autorefresh process by
   reducing the number of queries to SortingHat.


## grimoire-elk 0.105.0 - (2023-05-17)

**Breaking changes:**

 * SortingHat configuration\
   When SortingHat's multi-tenancy is enabled, the tenant is included in
   all clients during execution.


  ## grimoire-elk 0.104.6 - (2023-04-28)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.104.5 - (2023-04-27)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.104.4 - (2023-04-26)
  
  * Update Poetry's package dependencies

## grimoire-elk 0.104.3 - (2023-04-21)

**Bug fixes:**

 * Fix typo in SortingHat fields\
   Some fields were not updated from the old version of SortingHat. Now
   the API returns 'isBot' and 'genderAcc' instead of 'is_bot' and
   'gender_acc'.

**Breaking changes:**

 * Define date format for Git documents\
   Git by default uses a custom date format that is not recognized by
   Elastic/OpenSearch.  Specify the format for the AuthorDate and
   CommitDate fields in Git documents to support the formats usually
   returned by `git log`.  This allows us to do proper filtering on these
   fields such as "all documents with CommitDate before 2020".
 * Update support for ES/OS versions > 7.X/1.X\
   Currently the support for ES/OS versions > 7.X/1.X is broken due to a
   faulty version check.  Add a static method to the Elasticsearch object
   that correctly checks whether the version is 'legacy' (defined as ES
   6.X or OS 0.X) and adds this check to various places in the codebase.
   Additionally, fixes an instance where `interval` is used instead of
   `fixed_interval`. `interval` was deprecated in ES7.


  ## grimoire-elk 0.104.2 - (2023-02-03)
  
  * Update Poetry's package dependencies

  ## grimoire-elk 0.104.1 - (2023-02-01)
  
  * Update Poetry's package dependencies

## grimoire-elk 0.104.0 - (2023-02-01)

**Bug fixes:**

 * SortingHatClientError exception fixed\
   Exception raised with SortingHatClientError has been fixed in
   add_identity method.
 * SortingHat mandatory parameters\
   Include SortingHat 0.8.x mandatory parameters. Now `sh_port`,
   `sh_path`, `sh_ssl` are required.

**Breaking changes:**

 * Support of SortingHat 0.8\
   GrimoireLab starts using a new version of SortingHat.  SortingHat is
   now a web service that provides an API to manage identities and
   organizations. GrimoireELK now connects to the new SortingHat database
   using GraphQL.  The previous version of SortingHat is now deprecated.


## grimoire-elk 0.103.3 - (2022-11-07)

**Bug fixes:**

 * Demographic study on Weblate fixed\
   Weblate demographic study is now updated to allow passing the study
   alias name by parameter.


## grimoire-elk 0.103.2 - (2022-10-31)

**New features:**

 * Development and reviewing contributors fields for Git items\
   New information about the participants during the development and the
   reviewing processes are available in `Git` items. The Git backend
   extracts these data from `Approved-by`, `Co-authored-by`,  and
   `Merged-by` trailers that some commit messages might include. For each
   one of these trailers, new fields will be available with the suffixes:
   `_bots`, `_domains`, `_names`, `_org_names`, and `_uuids`. Also for
   each one of these suffixes, there will be extra fields with the prefix
   `non_authored_`. This is an example of the fields available for the
   the trailer `Merged-by`:     - merged_by_multi_bots     -
   merged_by_multi_domains     - merged_by_multi_names     -
   merged_by_multi_org_names     - merged_by_multi_uuids     -
   non_authored_merged_by_multi_bots     -
   non_authored_merged_by_multi_domains     -
   non_authored_merged_by_multi_names     -
   non_authored_merged_by_multi_org_names
 * Option 'to-date' on the backend sections\
   The option `to-date` on the `setup.cfg` allows fetching data until a
   specific date. By default, this value is `2100-01-01`.

**Bug fixes:**

 * Onion study on Github pull requests fixed\
   Github pull requests onion study is now updated to allow passing the
   study alias name by parameter.
 * Onion study on Gitlab fixed\
   Gitlab onion study is now updated to allow passing the study alias
   name by parameter.

**Dependencies updateds:**

 * New Pandas version\
   Update Pandas to the newest version compatible with Python 3.7+.


## grimoire-elk 0.103.0 - (2022-09-26)

**New features:**

 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.10 - (2022-09-26)

**New features:**

 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.9 - (2022-09-26)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.8 - (2022-09-26)

**New features:**

 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.7 - (2022-09-23)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.6 - (2022-09-23)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.5 - (2022-09-23)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.4 - (2022-09-07)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * git-branches study skip uncloned repo\
   The git-branches study skips the repository when it is not cloned and
   continues with the next one.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.3 - (2022-08-23)

**New features:**

 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.
 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.2 - (2022-07-22)

**New features:**

 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```
 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## grimoire-elk 0.103.0-rc.1 - (2022-07-21)

**New features:**

 * Entry points to support enriching plugin mechanism\
   This plugin mechanism allows ELK to use third-party enriching
   backends. Developers only need to include the next snippet in their
   package configuration and ELK will import them as any other core
   backend.  ``` entry_points={"grimoire_elk": "mybackend =
   package.subpackage:get_connectors"}, ``` **NOTE**: In the previous
   example, `get_connectors` is a function that returns the backend
   classes needed to enrich a datasource. Take the next snipped as an
   example: ``` def get_connectors():     return {"backend": [MyBackend,
   MyBackendOcean, MyBackendEnrich, MyBackendCommand]} ```
 * Configurable study aliases\
   Before this fix, the name of the study aliases was hard coded. From
   now on, these values can be configured by passing them as parameters.

**Bug fixes:**

 * [raw/slack] Handle fields in "data.file"\
   Avoid the 'Failed to insert data to ES' error when a document contains
   at least one immense term in 'data.file' (whose UTF8 encoding is
   longer than the max length 32766).


## GrimoireELK 0.102.0 - (2022-06-24)

**Bug fixes:**

 * Poetry build version updated\
   The current version of the `poetry` builder was old and caused some
   problems when installing packages in developer mode. With this new
   version, it's possible to install ELK in developer or editable mode
   without using `setuptools` files (i.e. `setup.cfg` and `setup.py`).

**Others:**

 * GrimoireLab dependencies update\
   Dependencies have been updated to use the latest versions of
   GrimoireLab packages:
     - grimoirelab-toolkit >= 0.3
     - perceval >= 0.19
     - perceval-mozilla >= 0.3
     - perceval-opnfv >=0.2
     - perceval-puppet >= 0.2
     - perceval-weblate >= 0.2
     - graal >= 0.3
     - cereslib >= 0.3


## GrimoireELK 0.101.1 - (2022-06-03)

**Bug fixes:**

 * Pair programming regular expression improvement\
   The regular expression to detect pair programming authors in git
   datasource has been improved adding more connectors. The following
   list shows the current connectors: - `[aA][nN][dD]` - `&` - `+`
 * Fix fetching authors with min and max dates\
   The old query only returns 10000 items due to ElasticSearch and
   OpenSearch now use `composite` aggregation to paginate all buckets.


## GrimoireELK 0.101.0 - (2022-06-03)

**New features:**

 * Anonymize parameters\
   A new API function `anonymize_params(params)` allows to anonymize a
   list of parameters that developers might consider secrets. It will
   only take effect for those parameters which their names are defined on
   `SECRET_PARAMETERS` constant.
 * Dependencies file for development\
   Include a new requirements.txt file to install the development version
   from the repositories.
 * Support to add identities to SortingHat from Enrich class\
   New methods `add_identities` and `add_identity` are available on
   `Enrich` class to add new identities to a `SortingHat` database.
   Libraries using `elk` won't need to use `sortinghat` library to store
   new identities while enriching items.
 * Collect and enrich confluence by spaces\
   Users can collect and enrich Confluence data from specific spaces
   only, adding the option `--spaces=[SPACE1, SPACE2, ...]` to their
   `projects.json`. The following example shows how to collect and to
   enrich data from only the `DEV` and `SUPP` spaces only. ``` {
   "project": {         "confluence": [             "http://example.com
   --spaces=[DEV, SUPP]"         ]     } } ```
 * Add demographic study to mattermost\
   Add the demographic study to the mattermost data source.

**Bug fixes:**

 * Skip sources with wrong arguments\
   Sources won't interrupt collection or enrichment processes when any of
   their arguments are invalid. Instead, they will be skipped.
 * Git fix 'is_git_commit_multi_author'\
   This fixes the `is_git_commit_multi_author` field when the commit has
   more than one author and `pair-programming` is enabled. This is
   important for calculating `pair_programming_*` values.
 * SortingHat as a required package\
   SortingHat is required for running grimoireelk but it was set as an
   extra dependency. From now on sortinghat is always installed with
   grimoireelk.
 * Confluence credentials not stored in raw indexes\
   Credentials for Confluence datasource were stored in raw indices as
   part of the URL. For now on, credentials that are part of URLs are
   removed before any data is stored.

**Feature removals:**

 * Drop Python 3.6 support\
   Python 3.6 reached the end of life at the end of 2021. This means it
   won't receive new updates or patches to fix security issues.
   Therefore, this package will only work with Python >= 3.7 from now on.
 * FINOS Meeting backend removed\
   FINOS Meeting backend is not supported anymore. The Perceval backend
   was archived in Bitergia repository so it shouldn't be part of the
   core anymore because it's software not maintained.


## GrimoireELK 0.75.0 - (2020-06-20)

**New features:**

 * Add support for RocketChat (#876)\
   Added support for collecting and enriching the messages of RocketChat
   instances. The schema of the enriched data has been included. Tests
   have been added too. This work is a community contribution, which has
   been possible thanks to Obaro Ikoh and Animesh Kumar.


## GrimoireELK 0.72.0 - (2020-04-24)

**New features:**

 * code complexity per language (#782)\
   Add language differentiation to code complexity analysis (Cocom)
 * Add support for Gitter (#820)\
   Added support for creating raw and enriched indexes of message from
   Gitter. The visualizations and tests for extracted data have also been
   added.
 * Add support for Pagure (#821)\
   Added support for creating raw and enriched indexes of Pagure. Issued
   from a Pagure repository are enriched. Additionally, the comments of
   these issues (if present) are also enriched. The schema for this
   backend has also been added. Tests have been included accordingly.
 * Store and process GitHub events\
   ELK is enhanced to process and store events collected from the GitHub
   API. The events currently supported are - LabeledEvent -
   UnlabeledEvent - AddedToProjectEvent - MovedColumnsInProjectEvent -
   RemovedFromProjectEvent - CrossReferencedEvent - ClosedEvent

**Bug fixes:**

 * Fix missing geolocation in Meetup event (#826)\
   This fix allows to process Meetup events for which geolocation
   information isn't provided. For these events, the corresponding
   attribute in the enriched index (`venue_geolocation`) is set to None.