Skip to content

Commit

Permalink
Fix falkordb memories + few small fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@YoanSallami
YoanSallami committed Sep 8, 2024
1 parent 0eb70f3 commit ff90ebd
Show file tree
Hide file tree
Showing 47 changed files with 2,896 additions and 3,025 deletions.
154 changes: 19 additions & 135 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@
![Beta](https://img.shields.io/badge/Release-Beta-blue)
[![License: GPL-3.0](https://img.shields.io/badge/License-GPL-green.svg)](https://opensource.org/license/gpl-3-0/)

<p align="center">
<img alt="HybridAGI long-term memory" src="img/memories.svg"/>
</p>

</div>

**Disclaimer:** We are currently refactoring the project for better modularity and better ease of use. For now, only the Local integration if available, the FalkorDB & Kuzu integration will be done at the end of this refactoring. At that time we will accept contributions for the integration of other Cypher-based graph databases. For more information, join the Discord channel.
Expand Down Expand Up @@ -64,122 +60,33 @@ HybridAGI is build upon years of experience in making reliable Robotics systems.

We provide everything for you to build your LLM application with a focus around Cypher Graph databases. We provide also a local database for rapid prototyping before scaling your application with one of our integration.

<div align="center">

![pipeline](img/memories.png)

</div>

### Predictable/Deterministic behavior and infinite number of tools

Because we don't let the Agent choose the sequence of tools to use, we can use an infinite number of tools. By following the Graph Programs, we ensure a predictable and deterministic methodology for our Agent system. We can combine every memory system into one unique Agent by using the corresponding tools without limitation.

```python
import hybridagi.core.graph_program as gp

main = gp.GraphProgram(
name="main",
description="The main program",
)

main.add(gp.Decision(
id="is_objective_unclear",
purpose="Check if the Objective's is unclear",
question="Is the Objective's question unclear?",
))

main.add(gp.Action(
id="clarify",
purpose="Ask one question to clarify the user's Objective",
tool="AskUser",
prompt="Please pick one question to clarify the Objective's question",
))

main.add(gp.Action(
id="answer",
purpose="Answer the question",
tool="Speak",
prompt="Please answer to the Objective's question",
))

main.add(gp.Action(
id="refine_objective",
purpose="Refine the objective",
tool="UpdateObjective",
prompt="Please refine the user Objective",
))

main.connect("start", "is_objective_unclear")
main.connect("is_objective_unclear", "clarify", label="Clarify")
main.connect("is_objective_unclear", "answer", label="Answer")
main.connect("clarify", "refine_objective")
main.connect("refine_objective", "answer")
main.connect("answer", "end")

main.build() # Verify the structure of the program

print(main)
# // @desc: The main program
# CREATE
# // Nodes declaration
# (start:Control {id: "start"}),
# (end:Control {id: "end"}),
# (is_objective_unclear:Decision {
# id: "is_objective_unclear",
# purpose: "Check if the Objective's is unclear",
# question: "Is the Objective's question unclear?"
# }),
# (clarify:Action {
# id: "clarify",
# purpose: "Ask one question to clarify the user's Objective",
# tool: "AskUser",
# prompt: "Please pick one question to clarify the Objective's question"
# }),
# (answer:Action {
# id: "answer",
# purpose: "Answer the question",
# tool: "Speak",
# prompt: "Please answer to the Objective's question"
# }),
# (refine_objective:Action {
# id: "refine_objective",
# purpose: "Refine the objective",
# tool: "UpdateObjective",
# prompt: "Please refine the user Objective"
# }),
# // Structure declaration
# (start)-[:NEXT]->(is_objective_unclear),
# (is_objective_unclear)-[:CLARIFY]->(clarify),
# (is_objective_unclear)-[:ANSWER]->(answer),
# (clarify)-[:NEXT]->(refine_objective),
# (answer)-[:NEXT]->(end),
# (refine_objective)-[:NEXT]->(answer)
```
<div align="center">

![pipeline](img/graph_program.png)

</div>

### Modular Pipelines

With HybridAGI you can build data extraction pipelines, RAG applications or advanced Agent systems, each being possibly optimized by using DSPy optimizers. We also provide pre-made modules and metrics for easy prototyping.

Each module and data type is *strictly typed and use Pydantic* as data validation layer. You can build pipelines in no time by stacking Modules sequentially like in Keras or HuggingFace.

```python
from hybridagi.embeddings import SentenceTransformerEmbeddings
from hybridagi.readers import PDFReader
from hybridagi.core.pipeline import Pipeline
from hybridagi.modules.splitters import DocumentSentenceSplitter
from hybridagi.modules.embedders import DocumentEmbedder

embeddings = SentenceTransformerEmbeddings(
model_name_or_path = "all-MiniLM-L6-v2",
dim = 384, # The dimention of the embeddings vector
)

reader = PDFReader()
input_docs = reader("data/SpelkeKinzlerCoreKnowledge.pdf") # This is going to extract 1 document per page

# Now that we have our input documents, we can start to make our data processing pipeline

pipeline = Pipeline()
<div align="center">

pipeline.add("chunk_documents", DocumentSentenceSplitter())
pipeline.add("embed_chunks", DocumentEmbedder(embeddings=embeddings))
![pipeline](img/pipeline.png)

output_docs = pipeline(input_docs)
```
</div>

### Native tools

Expand Down Expand Up @@ -210,34 +117,11 @@ We provide the following list of native tools to R/W into the memory system or m

You can add more tools by using the `FunctionTool` and python functions like nowadays function calling.

```python
import requests
from hybridagi.modules.agents.tools import FunctionTool

# The function inputs should be one or multiple strings, you can then convert or process them in your function
# The docstring and input arguments will be used to create automatically a DSPy signature
def get_crypto_price(crypto_name: str):
"""
Please only give the name of the crypto to fetch like "bitcoin" or "cardano"
Never explain or apology, only give the crypto name.
"""
base_url = "https://api.coingecko.com/api/v3/simple/price?"
complete_url = base_url + "ids=" + crypto_name + "&vs_currencies=usd"
response = requests.get(complete_url)
data = response.json()

# The output of the tool should always be a dict
# It usually contains the sanitized input of the tool + the tool result (or observation)
if crypto_name in data:
return {"crypto_name": crypto_name, "result": str(data[crypto_name]["usd"])+" USD"}
else:
return {"crypto_name": crypto_name, "result": "Invalid crypto name"}

my_tool = FunctionTool(
name = "GetCryptoPrice",
func = get_crypto_price,
)
```
<div align="center">

![pipeline](img/custom_tool.png)

</div>

### Graph Databases Integrations

Expand Down
35 changes: 21 additions & 14 deletions hybridagi/core/datatypes.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,12 @@ class Document(BaseModel):
parent_id: Optional[Union[UUID, str]] = Field(description="Identifier for the parent document", default=None)
vector: Optional[List[float]] = Field(description="Vector representation of the document", default=None)
metadata: Optional[Dict[str, Any]] = Field(description="Additional information about the document", default={})
created_at: datetime = Field(description="Time when the document was created", default_factory=datetime.now)

def to_dict(self):
return {"text": self.text, "metadata": self.metadata}
if self.metadata:
return {"text": self.text, "metadata": self.metadata}
else:
return {"text": self.text}

class DocumentList(BaseModel, dspy.Prediction):
docs: Optional[List[Document]] = Field(description="List of documents", default=[])
Expand Down Expand Up @@ -74,13 +76,18 @@ class Entity(BaseModel):
description: Optional[str] = Field(description="Description of the entity", default=None)
vector: Optional[List[float]] = Field(description="Vector representation of the document", default=None)
metadata: Optional[Dict[str, Any]] = Field(description="Additional information about the document", default={})
created_at: datetime = Field(description="Time when the entity was created", default_factory=datetime.now)

def to_dict(self):
if self.description is not None:
return {"name": self.name, "label": self.label, "description": self.description, "metadata": self.metadata}
if self.metadata:
if self.description is not None:
return {"name": self.name, "label": self.label, "description": self.description, "metadata": self.metadata}
else:
return {"name": self.name, "label": self.label, "metadata": self.metadata}
else:
return {"name": self.name, "label": self.label, "metadata": self.metadata}
if self.description is not None:
return {"name": self.name, "label": self.label, "description": self.description}
else:
return {"name": self.name, "label": self.label}

class EntityList(BaseModel, dspy.Prediction):
entities: List[Entity] = Field(description="List of entities", default=[])
Expand Down Expand Up @@ -108,10 +115,12 @@ class Relationship(BaseModel):
name: str = Field(description="Relationship name")
vector: Optional[List[float]] = Field(description="Vector representation of the relationship", default=None)
metadata: Optional[Dict[str, Any]] = Field(description="Additional information about the relationship", default={})
created_at: datetime = Field(description="Time when the relationship was created", default_factory=datetime.now)

def to_dict(self):
return {"name": self.name, "metadata": self.metadata}
if self.metadata:
return {"name": self.name, "metadata": self.metadata}
else:
return {"name": self.name}

class Fact(BaseModel):
id: Union[UUID, str] = Field(description="Unique identifier for the fact", default_factory=uuid4)
Expand All @@ -121,7 +130,6 @@ class Fact(BaseModel):
weight: float = Field(description="The fact weight (between 0.0 and 1.0, default 1.0)", default=1.0)
vector: Optional[List[float]] = Field(description="Vector representation of the fact", default=None)
metadata: Optional[Dict[str, Any]] = Field(description="Additional information about the fact", default={})
created_at: datetime = Field(description="Time when the fact was created", default_factory=datetime.now)

def to_cypher(self) -> str:
if self.subj.description is not None:
Expand All @@ -146,7 +154,10 @@ def from_cypher(self, cypher_fact:str, metadata: Dict[str, Any] = {}) -> "Fact":
raise ValueError("Invalid Cypher fact provided")

def to_dict(self):
return {"fact": self.to_cypher(), "metadata": self.metadata}
if self.metadata:
return {"fact": self.to_cypher(), "metadata": self.metadata}
else:
return {"fact": self.to_cypher()}

class FactList(BaseModel, dspy.Prediction):
facts: List[Fact] = Field(description="List of facts", default=[])
Expand Down Expand Up @@ -245,7 +256,6 @@ class UserProfile(BaseModel):
profile: Optional[str] = Field(description="The user profile", default="An average user")
vector: Optional[List[float]] = Field(description="Vector representation of the user", default=None)
metadata: Optional[Dict[str, Any]] = Field(description="Additional information about the user", default={})
created_at: datetime = Field(description="Time when the user profile was created", default_factory=datetime.now)

def to_dict(self):
return {"name": self.name, "profile": self.profile, "metadata": self.metadata}
Expand Down Expand Up @@ -318,9 +328,6 @@ class AgentStep(BaseModel):
parent_id: Optional[Union[UUID, str]] = Field(description="The previous step id if any", default=None)
hop: int = Field(description="The step hop", default=0)
step_type: AgentStepType = Field(description="The step type")
weight: float = Field(description="The step weight (between 0.0 and 1.0, default 1.0)", default=1.0)
name: Optional[str] = Field(description="The name of the step", default=None)
description: Optional[str] = Field(description="The description of the step", default=None)
inputs: Optional[Dict[str, Any]] = Field(description="The inputs of the step", default=None)
outputs: Optional[Dict[str, Any]] = Field(description="The outputs of the step", default=None)
vector: Optional[List[float]] = Field(description="Vector representation of the step", default=None)
Expand Down
2 changes: 1 addition & 1 deletion hybridagi/core/graph_program.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ def to_cypher(self):
return cypher

def to_dict(self):
return {"name": self.name, "description": self.description, "routine": self.to_cypher()}
return {"name": self.name, "routine": self.to_cypher()}

def save(self, folderpath: str = ""):
"""
Expand Down
2 changes: 1 addition & 1 deletion hybridagi/memory/document_memory.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
class DocumentMemory(ABC):

@abstractmethod
def exist(self, doc_id) -> bool:
def exist(self, doc_id: Union[UUID, str]) -> bool:
raise NotImplementedError(
f"DocumentMemory {type(self).__name__} is missing the required 'exist' method."
)
Expand Down
2 changes: 1 addition & 1 deletion hybridagi/memory/fact_memory.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
class FactMemory(ABC):

@abstractmethod
def exist(self, entity_or_fact_id) -> bool:
def exist(self, entity_or_fact_id: Union[UUID, str]) -> bool:
raise NotImplementedError(
f"FactMemory {type(self).__name__} is missing the required 'exist' method."
)
Expand Down
Loading

0 comments on commit ff90ebd

Please sign in to comment.