Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modified readme file #742

Closed
wants to merge 4 commits into from
Closed

Modified readme file #742

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c0ca29d
Issue KN-923 feat: Added Transaction event processor job
SurabhiAngadi Sep 5, 2023
f9e6230
Issue KN-923 feat: Added transaction-event-processor job
SurabhiAngadi Sep 5, 2023
9f0d26e
Issue KN-923: removed duplicates
SurabhiAngadi Sep 5, 2023
08ffcb7
Issue KN-923: Modified README.md file
SurabhiAngadi Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 6 additions & 55 deletions jobs-distribution/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,6 @@
<artifactId>flink-streaming-scala_${scala.version}</artifactId>
<version>${flink.version}</version>
<type>jar</type>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>activity-aggregate-updater</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>relation-cache-updater</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>post-publish-processor</artifactId>
Expand All @@ -44,12 +31,6 @@
<version>1.0.0</version>
<type>jar</type>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>questionset-publish</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>content-publish</artifactId>
Expand All @@ -62,36 +43,12 @@
<version>1.0.0</version>
<type>jar</type>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>enrolment-reconciliation</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>collection-cert-pre-processor</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>collection-certificate-generator</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>asset-enrichment</artifactId>
<version>1.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>audit-history-indexer</artifactId>
<version>1.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>auto-creator-v2</artifactId>
Expand All @@ -104,24 +61,12 @@
<version>1.0.0</version>
<type>jar</type>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.sunbird</groupId>-->
<!-- <artifactId>metrics-data-transformer</artifactId>-->
<!-- <version>1.0.0</version>-->
<!-- <type>jar</type>-->
<!-- </dependency>-->
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>qrcode-image-generator</artifactId>
<version>1.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>audit-event-generator</artifactId>
<version>1.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>dialcode-context-updater</artifactId>
Expand Down Expand Up @@ -152,6 +97,12 @@
<version>1.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.sunbird</groupId>
<artifactId>transaction-event-generator</artifactId>
<version>1.0.0</version>
<type>jar</type>
</dependency>
</dependencies>

<build>
Expand Down
10 changes: 2 additions & 8 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
<maven>3.0.0</maven>
</prerequisites>

<groupId>org.sunbird</groupId>
<groupId>org.sunbird</groupId>
<artifactId>knowledge-platform-jobs</artifactId>
<version>1.0</version>
<packaging>pom</packaging>
Expand All @@ -27,18 +27,12 @@
</activation>
<modules>
<module>jobs-core</module>
<!-- <module>relation-cache-updater</module>-->
<!-- <module>activity-aggregate-updater</module>-->
<module>post-publish-processor</module>
<!-- <module>credential-generator</module>-->
<module>publish-pipeline</module>
<module>search-indexer</module>
<!-- <module>enrolment-reconciliation</module>-->
<module>auto-creator-v2</module>
<module>content-auto-creator</module>
<module>audit-history-indexer</module>
<module>audit-event-generator</module>
<!-- <module>metrics-data-transformer</module>-->
<module>transaction-event-processor</module>
<module>qrcode-image-generator</module>
<module>dialcode-context-updater</module>
<module>cassandra-data-migration</module>
Expand Down
65 changes: 65 additions & 0 deletions transaction-event-processor/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Transaction Event Processor

Transaction Event Processor job generates an audit event for every transaction in Graph updation for content data modification. The job utilizes the neo4j mutation data to create an index of transactions for audit purposes. For each neo4j transaction, the job will audit the old and new values of the updated object.

## Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a yarn or kubernetes.
### Prerequisites

1. Download flink-1.13.6-scala_2.12 from [apache-flink-downloads](https://www.apache.org/dyn/closer.lua/flink/flink-1.13.6/flink-1.13.6-bin-scala_2.12.tgz).
2. Download [hadoop dependencies](https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar) (only for running on Yarn). Copy the hadoop dependency jar under lib folder of the flink download.
3. export HADOOP_CLASSPATH=`<hadoop-executable-dir>/hadoop classpath` either in .bashrc or current execution shell.
4. Docker installed.
5. A running yarn cluster or a kubernetes cluster.

### Build

mvn clean install

## Deployment

### Yarn

Flink requires memory to be allocated for both job-manager and task manager. -yjm parameter assigns job-manager memory and -ytm assigns task-manager memory.

```
./bin/flink run -m yarn-cluster -p 2 -yjm 1024m -ytm 1024m <knowledge-platform-jobs>/transaction-event-processor/target/transaction-event-processor-0.0.1.jar
```

### Kubernetes

```
# Create a single node cluster
k3d create --server-arg --no-deploy --server-arg traefik --name flink-cluster --image rancher/k3s:v1.0.0
# Export the single node cluster into KUBECONFIG in the current shell or in ~/.bashrc.
export KUBECONFIG="$(k3d get-kubeconfig --name='flink-cluster')"

# Only for Mac OSX
# /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl
rm /usr/local/bin/kubectl
brew link --overwrite kubernetes-cli

# Create a configmap using the flink-configuration-configmap.yaml
kubectl create -f knowledge-platform-job/kubernetes/flink-configuration-configmap.yaml

# Create pods for jobmanager-service, job-manager and task-manager using the yaml files
kubectl create -f knowledge-platform-job/kubernetes/jobmanager-service.yaml
kubectl create -f knowledge-platform-job/kubernetes/jobmanager-deployment.yaml
kubectl create -f knowledge-platform-job/kubernetes/taskmanager-deployment.yaml

# Create a port-forwarding for accessing the job-manager UI on localhost:8081
kubectl port-forward deployment/flink-jobmanager 8081:8081

# Submit the job to the Kubernetes single node cluster flink-cluster
./bin/flink run -m localhost:8081 <knowledge-platform-job>/transaction-event-processor/target/transaction-event-processor-0.0.1.jar

# Commands to delete the pods created in the cluster
kubectl delete deployment/flink-jobmanager
kubectl delete deployment/flink-taskmanager
kubectl delete service/flink-jobmanager
kubectl delete configmaps/flink-config

# Command to stop the single-node cluster
k3d stop --name="flink-cluster"
```
Loading
Loading