before-refactor

What's Changed

• Azure terraform changes by @ManojKrishnaChintaluri in #150
• Local Installation Scripts by @ravismula in #151
• terraform destroy deletes cloudwatch log group by @divyagovindaiah in #152
• Add connector info into system events ingestion spec. by @sowmya-dixit in #153
• #8: Update Documentation for AWS and Azure by @ManojKrishnaChintaluri in #154
• Correct the redis ip address in the superset helm chart by @divyagovindaiah in #157
• Issue #304 postgres changes in datasource tables to support Hudi schema by @GayathriSrividya in #156
• Revert "Issue #304 postgres changes in datasource tables to support Hudi schema" by @GayathriSrividya in #159
• Release 1.0.5-GA by @ravismula in #162
• oss unified helm fix by @ravismula in #163
• Release 1.0.5-GA by @ravismula in #164
• Release v1.0.5-GA by @ravismula in #165
• Datasource DB schema changes to include type. by @sowmya-dixit in #169
• Terraform changes in obsrv to support lakehouse changes by @GayathriSrividya in #170
• Open sourcing transformer job by @SanthoshVasabhaktula in #173
• Merging Develop into Main by @GayathriSrividya in #171

New Contributors

• @SanthoshVasabhaktula made their first contribution in #173

Full Changelog: 1.0.4-GA...before-refactor

v1.0.4-GA

Release 1.0.4-GA (#155)

* #0000 feat azure terraform changes (#150)

* Local Installation Scripts (#151)

* Sanketika-Obsrv/issue-tracker#26: Helm Changes for Local Setup

* Sanketika-Obsrv/issue-tracker#26: CRD's for Local Setup

* delete cloudwatch loggroups when using terraform destroy

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Add connector info into system events ingestion spec. (#153)

* #8: Update Documentation for AWS and Azure (#154)

* Sanketika-Obsrv/issue-tracker#8: doc: update shell script documentation for infra setup

* Sanketika-Obsrv/issue-tracker#8: doc: update azure documentation

* Sanketika-Obsrv/issue-tracker#8: doc: update AWS documentation for velero backup

* Sanketika-Obsrv/issue-tracker#8: fix: typo in installation doc

---------

Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com>
Co-authored-by: divyagovindaiah <110388603+divyagovindaiah@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>

v1.0.3-GA

Release 1.0.3-GA (#148)

* Fix: removed docker secrets

* updated the grafana_secrete

* Feat: updated image tags according to release 1.1.0

* Issue #171 feat: update job to deployment for jobmanagers

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* develop into release 1.3.0 (#97)

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

---------

Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>

* #126 feat: add new property to datasources table (#99)

* #126 feat: add new property to datasources table

* #126 fix: allow property to be nullable

* #0000 - feat: Fix masterdata stats ingestion spec (#100)

* #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* #0000 - feat: Flink helm configuration changes (#103)

* #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #0000 - feat: Secor changes for unified failed topic (#104)

* feat: azure terraform changes to provision cluster (#106)

* feat: azure terraform changes to provision cluster

* cleanup azure terraform code

* update node count and instance size for azure (#107)

* Issue #305 feat: exclude system-events for the datasource validation (#108)

* Issue #0000 fix: system events ingestion spec typo fix (#105)

* Issue #0000 fix: system events ingestion spec typo fix

* Issue #0000 fix: system-events ingestion spec update to add dataset_type

* #0 fix: add unique relation for source config table (#110)

* feat new image tags (#112)

* * #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* * #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #000: Rebased the develop with release-1.0.0-GA

* #0000 - feat: Import error dashboard (#119)

* [ Secor Timzone ] - Infra automation changes to take the timezone property (#120)

* #000: Configuring the helm to handle the instance level timezone

* Removed the unwanted timezone properties

* #203: Removed the commented code

* #226 Feat: Update postgres migrations to add system settings and default values (#121)

* #226 fix: add system_settings table

* #226 fix: Add data encryption key to PostgreSQL migration

* #226 fix: move system settings new migration script

* update tags for ga release and fix undeclared variable in secor (#122)

* System Telemetry Events Ingestion Spec and Backup Configurations (#118)

* #223: Configured the auto submission of telemetry events ingestion spec and configured the secor backup for system telemetry events

* #223: Updated the secor jobs variabl by including system-telemetry-events process

* #203: typo error fix

* #203: Configured the flag to submit the ingestion spec

* #203: Configured the flag to submit the ingestion spec

* #67 fix: add grant permissions for created table (#123)

* #172 feat: add outputs for kafka service (#130)

* #172 feat: add outputs for kafka service (#132)

* #245 feat: Update Druid, PostgreSQL, and Superset versions (#133)

* #0 feat: Update Druid, PostgreSQL, and Superset versions

* #245 fix: Grant all privileges on schema to superset and druid_raw

* #245 fix: Update database ownership in migration scripts

* #241 Enhance Infrastructure Flexibility, Deployment Reliability, and Observability (#131)

* #241:feat: Infra automation script

* #241 feat: Toggle vpc creation, Toggle velero user creation

* update: #241 main.tf for velero user creation

* remove: #241 revert automation script changes

* update: #241 Toggle to change service type and dataset api cm bug fix

* #241:feat: Updated the infra setup shell script

* #241:doc: Updated the setup doc file

* #241:doc: Updated the setup doc file and removed commented code

* #241:doc: Added the Prerequisites section

* feat: #241 Toggle to enable cloudwatch logs for EKS

* feat: #241 Deployment changes to provision infra

* feat: # 241 Deployment script for provisioning infra

* clean: #241 remove unwanted code.

* update: #241 update helm version in shell script and remove overrides.tfvars file

* enhance: #241 Enhance shell script create a tfvars from setup.conf file

* fix: #241 typo in displayed message, fix terraform installation command

* update: #241 eks cluster version

* update: #241 kuberentes service type

* update: #241 kubernetes service type to loadbalancer

* Update helm version

---------

Co-authored-by: Manjunath Davanam <manjunath@sanketika.in>

* update: AKS cluster version (#134)

* bug fix: #241 region typo and modify shell script (#135)

* bug fix: #241 region typo and modify shell script

* Bug fix: #241 modify velero module

* Installation fixes (#138)

* fix: installation-fixes and enhancements

* update: [1/2] Enabling cluster logs by default. [2/2] Creating vpc, kong ingress, velero user by default

* feat: Enable kafka connector and fix class name for kafka connector

* update: add decrypt command

* upgrade: eks version

* dummy commit to reflect changes

* dummy commit to reflect changes

* modify desired node capacity

* revert node capacity changes

* #0000 upgrade eks add ons (#144)

* #0000 update granularity for system events and system telemetry events (#145)

* #0000 update granularity for system events and system telemetry events

* code refactoring

Signed-off-by: SurabhiAngadi <surabhia@sanketika.in>

* code refactoring

Signed-off-by: SurabhiAngadi <surabhia@sanketika.in>

* feat: added cluster-overrides.tfvars

Signed-off-by: SurabhiAngadi <surabhia@sanketika.in>

---------

Signed-off-by: SurabhiAngadi <surabhia@sanketika.in>
Co-authored-by: SurabhiAngadi <surabhia@sanketika.in>

* fix: Update default max.message.bytes configuration of kafka topic to 10mb. (#147)

* Installation fixes (#146)

* #000:feat: Updated the obsrv automation script

* #000:feat: Updated the obsrv automation script

* #000:feat: Updated the configurations

* #00 fix: update gcp automation scripts

* #00 fix: update gcp installation instructions

* feat: azure os helm chart

* #0000 update aks node count.

---------

Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com>
Co-authored-by: ManojKrishnaChintaluri <manojc@sanketika.in>

---------

Signed-off-by: SurabhiAngadi <surabhia@sanketika.in>
Co-authored-by: GayathriSrividya <gayathrirajavarapu@sanketika.in>
Co-authored-by: divyagovindaiah <divyag@sanketika.in>
Co-authored-by: Manjunath Davanam <manjunath@sanketika.in>
Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>
Co-authored-by: Anand Parthasarathy <anandp504@gmail.com>
Co-authored-by: Ravinder Kumar <yravinderkumar33@gmail.com>
Co-authored-by: SurabhiAngadi <surabhia@sanketika.in>
Co-authored-by: ManojKrishnaChintaluri <manojc@sanketika.in>

v1.0.2-GA

What's Changed

• Release-1.0.2-GA to main by @manjudr in #141
• #0000 update release tags by @ManojKrishnaChintaluri in #142
• #0000 update image tags for 1.0.2-GA release by @ManojKrishnaChintaluri in #143

Full Changelog: 1.0.1-GA...1.0.2-GA

v1.0.1-GA

Develop to release-1.0.1-GA (#136) (#137)

v1.3.1

1.3.1 Release fixes (#115)

* Fix: removed docker secrets

* updated the grafana_secrete

* Feat: updated image tags according to release 1.1.0

* Issue #171 feat: update job to deployment for jobmanagers

* develop into release 1.3.0 (#97)

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

---------

Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>

* Release 1.3.1 Changes (#109)

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #126 feat: add new property to datasources table (#99)

* #126 feat: add new property to datasources table

* #126 fix: allow property to be nullable

* #0000 - feat: Fix masterdata stats ingestion spec (#100)

* #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* #0000 - feat: Flink helm configuration changes (#103)

* #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #0000 - feat: Secor changes for unified failed topic (#104)

* feat: azure terraform changes to provision cluster (#106)

* feat: azure terraform changes to provision cluster

* cleanup azure terraform code

* update node count and instance size for azure (#107)

* Issue #305 feat: exclude system-events for the datasource validation (#108)

* Issue #0000 fix: system events ingestion spec typo fix (#105)

* Issue #0000 fix: system events ingestion spec typo fix

* Issue #0000 fix: system-events ingestion spec update to add dataset_type

* #0 fix: add unique relation for source config table (#110)

---------

Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>
Co-authored-by: Ravi Mula <ravishankarmula@gmail.com>
Co-authored-by: Anand Parthasarathy <anandp504@gmail.com>
Co-authored-by: Ravinder Kumar <yravinderkumar33@gmail.com>

* Merge changes to release (#113)

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #126 feat: add new property to datasources table (#99)

* #126 feat: add new property to datasources table

* #126 fix: allow property to be nullable

* #0000 - feat: Fix masterdata stats ingestion spec (#100)

* #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* #0000 - feat: Flink helm configuration changes (#103)

* #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #0000 - feat: Secor changes for unified failed topic (#104)

* feat: azure terraform changes to provision cluster (#106)

* feat: azure terraform changes to provision cluster

* cleanup azure terraform code

* update node count and instance size for azure (#107)

* Issue #305 feat: exclude system-events for the datasource validation (#108)

* Issue #0000 fix: system events ingestion spec typo fix (#105)

* Issue #0000 fix: system events ingestion spec typo fix

* Issue #0000 fix: system-events ingestion spec update to add dataset_type

* #0 fix: add unique relation for source config table (#110)

* feat new image tags (#112)

---------

Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>
Co-authored-by: Ravi Mula <ravishankarmula@gmail.com>
Co-authored-by: Anand Parthasarathy <anandp504@gmail.com>
Co-authored-by: Ravinder Kumar <yravinderkumar33@gmail.com>

* feat add kafka-connector release

* fixes

* [1/3] change instance from spot to on_demand [2/3] remove unwanted jobs [3/3] modify submit ingestion configmap

* #0 fix: update druid configurations and disable kafka-connector

* druid configuration changes for 30 workers

---------

Co-authored-by: GayathriSrividya <gayathrirajavarapu@sanketika.in>
Co-authored-by: divyagovindaiah <divyag@sanketika.in>
Co-authored-by: Manjunath Davanam <manjunath@sanketika.in>
Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>
Co-authored-by: Ravi Mula <ravishankarmula@gmail.com>
Co-authored-by: Anand Parthasarathy <anandp504@gmail.com>
Co-authored-by: Ravinder Kumar <yravinderkumar33@gmail.com>

v1.3.0

develop into release 1.3.0 (#97)

* #90 fix: Add default value for published_date in tables (#94)

* fix: remove restart policy from flink deployment manifest (#96)

* #196 Fix:Remove default region. (#95)

* fix: #196 remove default value for region

* update: #196 missing variable in variables.tf file

* #196 update:  example on how provide region

* #196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

* #58 feat: Separation of redis into dedup & meta/denorm.

---------

Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com>
Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com>
Co-authored-by: Sowmya N Dixit <sowmyadixit7@gmail.com>