Skip to content

just for sake of midterm ovulation will add my progress here.

Notifications You must be signed in to change notification settings

NEGRAH-S/ES_C4GT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

About

  • This repo will conatain all the task I did on weekly basis as my task for now isn't directly on parent code base.
  • Therefore, wont be having any major PR for the same .

Milestones decided

  • Selecting a remote File and doing all the task for prog=ject over it
  • Mapping of Json file made
  • [] kafka ingestion of ES
  • [] Mapping file for telementry
  • [] Flink job conversion
  • [] Automation and Documentation

Review schedule

  • First review on sharing of mapping
  • [] Second review complete the logic to ingest the data to es from kafka
  • [] Third review create the es mapping file for the sunbird telemetry and ingestion of the sunbird telemetry from kafka to es
  • [] Fourth review - Converting the algorithm to flink job.
  • [] Fifth review - Automation
  • [] Six review - Documentation

Phases

Phase - 1

  • RUN The Elastic search in local
  • Creation ES Mapping file
  • Submit the es mapping file
  • Create a sample data wchich is associated to es mapping
  • Submit the sample data

Phase -2

  • Run the kafka in locally
  • Create the topics
  • Push the events to kafka topic
  • Write a script or use any lib to index the data from the kafka to es

Phase - 3

  • Use the sunbird telemetry data to index the data from kafka to es
  • Create the es mapping file for the sunbird telemetry
  • Sample data from kafka to es

Phase - 4

  • Convert the script to flink job
  • Verifiy the functionality (kafka to es index) with sunbird telemetry

Phase - 5

  • Automate the es provision and run the es in the kubernetes
  • Run the flink in the kubernets
  • Create a helm charts for both to run them in the kubernetes

Phase - 6

  • Raise the final pull request to opensource with the documentation and testcases

About

just for sake of midterm ovulation will add my progress here.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages