Nebula #1265

jeanyu-habana · 2025-02-06T21:57:04Z

Description

support NebulaGraph cluster deployment in k8s

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

n/a

Tests

Tested the deployment and config scripts. NebulaGraph services were up and running. DB spaces can be created and managed.

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

get the latest

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

for more information, see https://pre-commit.ci

comps/third_parties/neo4j/deployment/kubernetes/README.md

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

xiguiw · 2025-02-25T13:37:22Z

@lianhao @yongfengdu
Would you please review this PR? Does the structure, style etc. confirm to ours?

lianhao · 2025-02-26T01:07:21Z

comps/third_parties/nebula/deployment/kubernetes/README.md

+```bash
+helm repo add nebula-operator https://vesoft-inc.github.io/nebula-operator/charts
+helm repo update
+helm install nebula-operator nebula-operator/nebula-operator --namespace=<namespace_name> --version=${chart_version}


We should either give specific chart version or let user to use the latest.

Okay, updated with a specific version.

lianhao · 2025-02-26T01:08:43Z

comps/third_parties/nebula/deployment/kubernetes/README.md

+## 1. Install cert-manager
+
+```bash
+kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml


Can we have cert-manager of versions othern than 1.9.1? I'm asking this because the user might have other versions of cert-manager in their cluster already.

good catch. I replaced specific version with <release_version> in the README to indicate that the user can use other versions. Thanks.

lianhao · 2025-02-26T01:10:46Z

comps/third_parties/nebula/deployment/kubernetes/README.md

+Run the following command to check the IP of the Service:
+
+```bash
+$ kubectl get service -l app.kubernetes.io/cluster=<nebula>  #<nebula> is a variable value. Replace it with the desired name.


what should be? Replace it by what value?

This can be any desired name.

lianhao · 2025-02-26T01:12:30Z

comps/third_parties/nebula/deployment/kubernetes/README.md

+Run the following command to connect to the NebulaGraph database using the IP of the <cluster-name>-graphd-svc Service above:
+
+```bash
+kubectl run -ti --image vesoft/nebula-console --restart=Never -- <nebula_console_name> -addr <cluster_ip>  -port <service_port> -u <username> -p <password>


It'd be better to give users direct bash commands to get <cluster_ip>, <service_port> .

Added the bash commands to find <cluster_ip>, <service_port> in README.

lianhao · 2025-02-26T01:13:43Z

comps/third_parties/nebula/deployment/kubernetes/community_edition.yaml

+      resources:
+        requests:
+          storage: 2Gi
+      storageClassName: "${STORAGE_CLASS_NAME}"


If nebula requires k8s to have a valid storage class, we'd better to say it clearly in README. Also if we follow the README strictly, this ${STORAGE_CLASS_NAME} dereference could cause the kubectl create -f enterprise_edition.yaml fail due to the mismatch of storageClassName. Does nebula operator support not setting specific storageClassName?

Yes, a storage class is required. The current readme me already mentioned that the storageClassName in the yaml file. I have add the following in the README to address the issue:
"Create a StorageClass or use an existing one in the k8s cluster. Edit the config with the proper storageClassName and run the command below".

lianhao · 2025-02-26T01:16:20Z

comps/third_parties/nebula/deployment/kubernetes/community_edition.yaml

+        cpu: "1"
+        memory: "1Gi"
+    replicas: 1
+    image: vesoft/nebula-graphd


should we pin the image tag instead of using latest image? Just in case future versions of nebula might not be compatible with this resource operator definition.

Okay, updated the yaml file with specific version for all related images.

yongfengdu · 2025-02-26T01:59:17Z

I would propose to add this (If necessary) together with you mentioned HybridRAG.
Otherwise, it's not clear why we need to add this components - Adding an open source GraphDB and use OEPA CI to test it.

xiguiw · 2025-02-26T04:02:02Z

@jeanyu-habana

There is a graphRAG component implemented by @rbrugaro

GenAIComps/comps/dataprep/src/integrations/neo4j_llamaindex.py

Line 82 in c70f868

class GraphRAGStore(Neo4jPropertyGraphStore):

It is integrated into data-prep, to improve the OPEA RAG through data-prep and retriever.

To bring the NebulaGraph in, I have some questions.

What's the difference (or gap) between NebulaGraph and the existed GraphRAG?
How the NebulaGraph being integrated into OPEA components or improve the OPEA components functionality?
Build an example with third party is not the targets of OPEA because there is no contribution from OPEA.
Could you integrate it with OPEA retriever/data-prep?

Thanks!

xiguiw · 2025-02-26T04:02:42Z

@jeanyu-habana
Would you please address the comments and questions? Thanks!

yongfengdu · 2025-02-26T04:45:17Z

@jeanyu-habana I was convinced that, in a sense of providing more components for OPEA project, so any users could utilize this components for new AI apps, it's ok to put these instructions for reference.
Because we don't have related helm charts for this in https://github.com/opea-project/GenAIInfra, the test logic would be different and you'll need to maintain it by yourself.

I would propose to add this (If necessary) together with you mentioned HybridRAG. Otherwise, it's not clear why we need to add this components - Adding an open source GraphDB and use OEPA CI to test it.

jeanyu-habana · 2025-02-26T16:16:18Z

@yongfengdu Thank you for meeting with me. Just to summarize, we went over the example in GenAIExamples and GenAIInfra (i.e., milvus), and I agreed when we implement our HybridRAG in GenAIExamples and GenAIInfra, we will follow the guidance. However, having nebulagraph k8s deployment in the GenAIComps repo still has value. The script here is tested and can be used to deploy nebulagraph as microservice in k8s. Please let me know if there is anything needed, and we will address them ASAP.

jeanyu-habana · 2025-02-26T17:16:32Z

@jeanyu-habana

There is a graphRAG component implemented by @rbrugaro

GenAIComps/comps/dataprep/src/integrations/neo4j_llamaindex.py

Line 82 in c70f868

class GraphRAGStore(Neo4jPropertyGraphStore):

It is integrated into data-prep, to improve the OPEA RAG through data-prep and retriever.

To bring the NebulaGraph in, I have some questions.

What's the difference (or gap) between NebulaGraph and the existed GraphRAG?

How the NebulaGraph being integrated into OPEA components or improve the OPEA components functionality?
Build an example with third party is not the targets of OPEA because there is no contribution from OPEA.
Could you integrate it with OPEA retriever/data-prep?

Thanks!

Existing GraphRAG in OPEA uses Neo4J graph database. Several of RAG related microservices we are about to submit to OPEA can use either Neo4J or NebulaGraph. NebulaGraph is distrubuted and much faster than Neo4J, especially during query and used in production environment. The difference between Neo4J and NebulaGraph can be found here: https://www.nebula-graph.io/posts/Neo4j_Alternative_Open-Source_Distributed_and_Lightning_Fast.
There are multi ways we can integrate NebulaGraph microservice into the new projects we are about to submit. We can also integrate it into the existing OPEA retriever/data-prep (as you suggested above).

jeanyu-habana · 2025-02-26T17:19:24Z

@jeanyu-habana I was convinced that, in a sense of providing more components for OPEA project, so any users could utilize this components for new AI apps, it's ok to put these instructions for reference. Because we don't have related helm charts for this in https://github.com/opea-project/GenAIInfra, the test logic would be different and you'll need to maintain it by yourself.

I would propose to add this (If necessary) together with you mentioned HybridRAG. Otherwise, it's not clear why we need to add this components - Adding an open source GraphDB and use OEPA CI to test it.

Agreed. Thank you. We will make sure at least one of our submissions will use Nebula and have OPEA CI test enabled.

jeanyu-habana · 2025-02-26T17:20:22Z

@jeanyu-habana Would you please address the comments and questions? Thanks!

Please see "quoted" reply in the context of your original question.

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

jeanyu-habana · 2025-03-02T00:09:59Z

@lianhao Could you please review the changes I made per your requests? Thank you.

joshuayao

@jeanyu-habana Thank you for your contribution! As per the OPEA contribution guidelines, new components are required to include Docker Compose deployment support. Could you please provide this to ensure compliance with the guidelines?

xiguiw · 2025-03-06T01:20:01Z

2. There are multi ways we can integrate NebulaGraph microservice into the new projects we are about to submit. We can also integrate it into the existing OPEA retriever/data-prep (as you suggested above).

Only Data-pre and retrieval interacts with graph DB. Maybe you can refer to Graph RAG to integrate NebulaGraph into data-prep/retrieval.

siddhivelankar23 · 2025-03-17T16:26:02Z

Docker support has been added for nebula as requested, please refer to PR 1396

lvliang-intel · 2025-03-18T06:35:22Z

tests/third_parties/test_third_parties_nebula.sh

+    echo "[ test create ] creating space.."
+    query="CREATE SPACE my_space(partition_num=10, replica_factor=1, vid_type=FIXED_STRING(32)); USE my_space; CREATE TAG person(name string, age int);"
+
+    create_response=$(kubectl run -ti --rm --image vesoft/nebula-console --restart=Never -- nebula-console -addr "$cluster_ip" -port 9669 -u root -p vesoft -e "$query" 2>&1)


How can other microservices access Nebula Database? Are there any software APIs or endpoints available? Using Nebula-Console is not good way for this purpose.

How can other microservices access Nebula Database? Are there any software APIs or endpoints available? Using Nebula-Console is not good way for this purpose.

@jeanyu-habana

Could you resolve this?

jeanyu-habana and others added 17 commits February 4, 2025 13:05

k8s deployment for neo4j

3270982

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

updated README and added 2 scripts

0011a66

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

fee5ffe

for more information, see https://pre-commit.ci

fixed hardcoded NS

e7868fd

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

Merge branch 'main' into neo4jk8s

296f3ed

Merge remote-tracking branch 'origin/main' into neo4jk8s

5ceb03b

get the latest

Merge remote-tracking branch 'origin/neo4jk8s' into neo4jk8s

8701aab

removed cluster specific configs

fb7658f

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

Merge branch 'main' into neo4jk8s

9c2ba3b

Merge remote-tracking branch 'origin/main' into neo4jk8s

aaa6d95

rename

4076e21

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

Merge remote-tracking branch 'origin/neo4jk8s' into neo4jk8s

d3b7003

Merge branch 'main' into neo4jk8s

a20e914

Merge remote-tracking branch 'origin/main' into neo4jk8s

67b2da9

Merge remote-tracking branch 'origin/neo4jk8s' into neo4jk8s

a00e886

use kubernetes instead of k8s in the path

23f1ab4

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

NebulaGraph k8s deploy

ea733b3

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

jeanyu-habana requested review from lvliang-intel, letonghan and XinyaoWa as code owners February 6, 2025 21:57

[pre-commit.ci] auto fixes from pre-commit.com hooks

62a70d8

for more information, see https://pre-commit.ci

letonghan reviewed Feb 7, 2025

View reviewed changes

comps/third_parties/neo4j/deployment/kubernetes/README.md Outdated Show resolved Hide resolved

jeanyu-habana and others added 8 commits February 7, 2025 15:23

k8s deployment for neo4j

cf0e0c1

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

updated README and added 2 scripts

55942e5

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

fixed hardcoded NS

320d11e

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

0a45ece

for more information, see https://pre-commit.ci

removed cluster specific configs

b097d2b

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

rename

c56a3c5

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

use kubernetes instead of k8s in the path

f35edea

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

NebulaGraph k8s deploy

2f01606

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

jeanyu-habana and others added 2 commits February 18, 2025 21:28

Merge branch 'main' into nebula

442e659

Merge branch 'main' into nebula

afbb45f

xiguiw requested review from lianhao and yongfengdu February 25, 2025 13:35

lianhao requested changes Feb 26, 2025

View reviewed changes

jeanyu-habana and others added 5 commits February 26, 2025 12:44

updates to address review comments

cb1535e

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

Merge remote-tracking branch 'origin/nebula' into nebula

ec70edb

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b7e24f

for more information, see https://pre-commit.ci

updated README with storageClass info

3dcba86

Signed-off-by: jeanyu-habana <jean1.yu@intel.com>

Merge remote-tracking branch 'origin/nebula' into nebula

aa4c672

jeanyu-habana requested a review from lianhao February 28, 2025 06:55

Merge branch 'main' into nebula

20480e4

joshuayao self-requested a review March 4, 2025 01:51

joshuayao requested changes Mar 4, 2025

View reviewed changes

siddhivelankar23 mentioned this pull request Mar 17, 2025

Docker support for nebula #1396

Merged

lvliang-intel reviewed Mar 18, 2025

View reviewed changes

joshuayao requested review from joshuayao and xiguiw March 25, 2025 01:03

Nebula #1265

Are you sure you want to change the base?

Nebula #1265

Uh oh!

Conversation

jeanyu-habana commented Feb 6, 2025

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

Uh oh!

xiguiw commented Feb 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yongfengdu commented Feb 26, 2025

Uh oh!

xiguiw commented Feb 26, 2025

Uh oh!

xiguiw commented Feb 26, 2025

Uh oh!

yongfengdu commented Feb 26, 2025

Uh oh!

jeanyu-habana commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeanyu-habana commented Feb 26, 2025

Uh oh!

jeanyu-habana commented Feb 26, 2025

Uh oh!

jeanyu-habana commented Feb 26, 2025

Uh oh!

jeanyu-habana commented Mar 2, 2025

Uh oh!

joshuayao left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiguiw commented Mar 6, 2025

Uh oh!

siddhivelankar23 commented Mar 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jeanyu-habana commented Feb 26, 2025 •

edited

Loading

joshuayao left a comment •

edited

Loading