Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 3.1 #1011

Merged
merged 42 commits into from
Sep 6, 2024
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
62cd6f6
Feature/orchestration v0 (#1006)
emrgnt-cmplxty Aug 30, 2024
44a741e
Feature/add update files workflow (#1010)
emrgnt-cmplxty Aug 30, 2024
fa88875
Feature/add enrichment flow (#1013)
emrgnt-cmplxty Aug 30, 2024
79eac6f
Feature/merged enrichment flow (#1016)
emrgnt-cmplxty Aug 30, 2024
cbe0f19
Rm graspologic (#1034)
shreyaspimpalgaonkar Sep 4, 2024
291e2d3
Feature/add hatchet api key setup rebased (#1040)
emrgnt-cmplxty Sep 4, 2024
5d3515e
Feature/nolan logs refactored (#1041)
emrgnt-cmplxty Sep 4, 2024
131c968
Pull open PRs into dev (#1042)
NolanTrem Sep 4, 2024
d280ce6
Add python files for templates in cli (#1043)
NolanTrem Sep 4, 2024
bb08b60
Merge branch 'main' into dev
NolanTrem Sep 4, 2024
b5fb31b
working hatchet integration (#1046)
emrgnt-cmplxty Sep 4, 2024
1fc3927
Update local_llm_neo4j_kg.toml
shreyaspimpalgaonkar Sep 4, 2024
ffab16a
Unstructured fixes (#1048)
shreyaspimpalgaonkar Sep 4, 2024
a46a638
Introduce File Provider (#1044)
NolanTrem Sep 4, 2024
3ff56d5
Make 7272 the default port (#1045)
NolanTrem Sep 4, 2024
39a4326
Fix poetry.lock
NolanTrem Sep 4, 2024
6fde6e6
Precommit
NolanTrem Sep 4, 2024
ef0b6fd
Enhance Dockerfile and add telemetry events (#1049)
shreyaspimpalgaonkar Sep 4, 2024
0be2767
Fix File Provider (#1050)
NolanTrem Sep 5, 2024
bdb9607
Feature/improve docs (#1051)
emrgnt-cmplxty Sep 5, 2024
c5bbdb3
fix compose
emrgnt-cmplxty Sep 5, 2024
bae8c0b
Add unstructured chunking configuration updates
shreyaspimpalgaonkar Sep 5, 2024
e0cb7e6
Revert "Add unstructured chunking configuration updates"
shreyaspimpalgaonkar Sep 5, 2024
90175de
Separate File Provider and Relational Database Provider (#1054)
NolanTrem Sep 5, 2024
91fcf70
undo changes in compose
NolanTrem Sep 5, 2024
3a4033c
Patch/fix unstructured config rebased (#1059)
emrgnt-cmplxty Sep 5, 2024
06db069
Graph docs (#1060)
shreyaspimpalgaonkar Sep 5, 2024
8a9eaf9
Remove duplicate method (#1061)
NolanTrem Sep 5, 2024
4634c63
update docs (#1064)
emrgnt-cmplxty Sep 6, 2024
1dc5043
rm extra prints
emrgnt-cmplxty Sep 6, 2024
8178d76
fix img
emrgnt-cmplxty Sep 6, 2024
208ad66
Fallback logic (#1062)
shreyaspimpalgaonkar Sep 6, 2024
511ec27
Refactor response models for clarity
shreyaspimpalgaonkar Sep 6, 2024
41be1a6
Refactor response types in router.
shreyaspimpalgaonkar Sep 6, 2024
4d463c2
Feature/fix agent (#1065)
emrgnt-cmplxty Sep 6, 2024
865e125
Patch/fix 123 (#1066)
emrgnt-cmplxty Sep 6, 2024
23855e7
Feature/add orchestration draft (#1067)
emrgnt-cmplxty Sep 6, 2024
7b955cb
Fix some of the tests (#1068)
NolanTrem Sep 6, 2024
ccb8170
Fix fallback parsing (#1069)
NolanTrem Sep 6, 2024
7700cc9
Feature/iterate on docs (#1070)
emrgnt-cmplxty Sep 6, 2024
a31ab8e
Fix restructuring enum (#1071)
NolanTrem Sep 6, 2024
5f79626
Feature/formatting cleanup (#1072)
emrgnt-cmplxty Sep 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/r2r-js-sdk-integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
POSTGRES_VECS_COLLECTION: ${{ secrets.POSTGRES_VECS_COLLECTION }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
r2r serve --port=8000 > r2r_server.log 2>&1 &
r2r serve --port=7272 > r2r_server.log 2>&1 &
sleep 15

- name: Use Node.js
Expand All @@ -54,7 +54,7 @@ jobs:

- name: Check if R2R server is running
run: |
curl http://localhost:8000/v2/health || echo "Server not responding"
curl http://localhost:7272/v2/health || echo "Server not responding"

- name: Display R2R server logs if server not responding
if: failure()
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ mintlify dev

### Publishing Changes

Install our Github App to auto propagate changes from your repo to your deployment. Changes will be deployed to production automatically after pushing to the default branch. Find the link to install on your dashboard.
Install our Github App to auto propagate changes from your repo to your deployment. Changes will be deployed to production automatically after pushing to the default branch. Find the link to install on your dashboard.

#### Troubleshooting

Expand Down
1 change: 0 additions & 1 deletion docs/api-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,3 @@ description: 'Build, scale, and manage user-facing Retrieval-Augmented Generatio
icon: 'message-code'
redirect: api-reference/introduction
---

2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/add_user_to_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Add User to Group'
openapi: 'POST /v2/add_user_to_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/agent.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Agent'
openapi: 'POST /v2/agent'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/analytics.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Analytics'
openapi: 'GET /v2/analytics'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/app_settings.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ title: 'App Settings'
openapi: 'GET /v2/app_settings'
---

Gets the latest settings for the application logic.
Gets the latest settings for the application logic.
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/assign_document_to_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Assign Document to Group'
openapi: 'POST /v2/assign_document_to_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/change_password.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Change Password'
openapi: 'POST /v2/change_password'
---
---
4 changes: 4 additions & 0 deletions docs/api-reference/endpoint/create_graph.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: 'Create Graph'
openapi: 'POST /v2/create_graph'
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/create_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Create Group'
openapi: 'POST /v2/create_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/delete.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Delete'
openapi: 'DELETE /v2/delete'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/delete_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Delete Group'
openapi: 'DELETE /v2/delete_group/{group_id}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/delete_user_info.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Delete User'
openapi: 'DELETE /v2/user'
---
---
4 changes: 2 additions & 2 deletions docs/api-reference/endpoint/document_chunks.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Document Chunks'
openapi: 'GET /v2/document_chunks'
---
openapi: 'GET /v2/document_chunks/{document_id}'
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/documents_overview.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Documents Overview'
openapi: 'GET /v2/documents_overview'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/enrich_graph.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'enrich_graph'
title: 'Enrich Graph'
openapi: 'POST /v2/enrich_graph'
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/get_document_groups.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Get Document Groups'
openapi: 'GET /v2/get_document_groups/{document_id}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/get_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Get Group'
openapi: 'GET /v2/get_group/{group_id}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/get_groups_for_user.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Get Groups for User'
openapi: 'GET /v2/get_groups_for_user/{user_id}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/get_user_info.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Get User'
openapi: 'GET /v2/user'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/get_users_in_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Get Users in Group'
openapi: 'GET /v2/get_users_in_group/{group_id}/{offset}/{limit}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/groups_overview.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Groups Overview'
openapi: 'GET /v2/groups_overview'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/health.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Health'
openapi: 'GET /v2/health'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/ingest_files.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Ingest Files'
openapi: 'POST /v2/ingest_files'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/list_groups.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'List Groups'
openapi: 'GET /v2/list_groups'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/login.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Login'
openapi: 'POST /v2/login'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/logout.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Logout'
openapi: 'POST /v2/logout'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/logs.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Logs'
openapi: 'GET /v2/logs'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/put_user_info.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Update User'
openapi: 'PUT /v2/user'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/rag.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'RAG'
openapi: 'POST /v2/rag'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/refresh_access_token.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Refresh Access Token'
openapi: 'POST /v2/refresh_access_token'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/register.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Register'
openapi: 'POST /v2/register'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/remove_document_from_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Remove Document from Group'
openapi: 'POST /v2/remove_document_from_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/remove_user_from_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Remove User from Group'
openapi: 'POST /v2/remove_user_from_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/request_password_reset.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Password Reset'
openapi: 'POST /v2/request_password_reset'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/score_completion.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Score Completion'
openapi: 'POST /v2/score_completion'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/search.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Search'
openapi: 'POST /v2/search'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/update_files.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Update Files'
openapi: 'POST /v2/update_files'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/update_group.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Update Group'
openapi: 'GET /v2/update_group'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/update_prompt.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Update Prompt'
openapi: 'POST /v2/update_prompt'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/users_overview.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Users Overview'
openapi: 'GET /v2/users_overview'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/endpoint/verify_email.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: 'Verify Email'
openapi: 'POST /v2/verify_email/{verification_code}'
---
---
2 changes: 1 addition & 1 deletion docs/api-reference/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ To get started with the R2R API, you'll need to:
1. Install R2R in your environment
2. Run the server with `r2r serve`, or customize your FastAPI for production settings.

For detailed installation and setup instructions, please refer to our [Installation Guide](/documentation/installation).
For detailed installation and setup instructions, please refer to our [Installation Guide](/documentation/installation).
2 changes: 1 addition & 1 deletion docs/api-reference/openapi.json

Large diffs are not rendered by default.

17 changes: 8 additions & 9 deletions docs/cookbooks/advanced-rag.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -100,16 +100,16 @@ The code above builds R2R with a custom factory which changes retrieval such tha

Now, to serve it in non-production settings, we can run the following:

```bash
```bash
python $r2r_workdir/py/core/examples/scripts/serve_with_hyde.py
```
This results in output like this -
This results in output like this -

```bash
2024-08-22 10:06:01,563 - INFO - core.providers.prompts.r2r_prompts - Loading prompts from SciPhi/r2r/py/core/providers/prompts/defaults
2024-08-22 10:06:01,584 - INFO - core.base.providers.prompt - Initializing PromptProvider with config extra_fields={} provider='r2r' default_system_name='default_system' default_task_name='default_rag' file_path=None.
2024-08-22 10:06:01,584 - INFO - core.base.providers.embedding - Initializing EmbeddingProvider with config extra_fields={} provider='litellm' base_model='openai/text-embedding-3-small' base_dimension=512 rerank_model=None rerank_dimension=None rerank_transformer_type=None batch_size=128 prefixes=None add_title_as_prefix=False concurrent_request_limit=256 max_retries=2 initial_backoff=1.0 max_backoff=60.0.
2024-08-22 10:06:01,584 - INFO - core.base.providers.llm - Initializing CompletionProvider with config: extra_fields={'concurrent_request_limit': 16} provider='litellm' generation_config=GenerationConfig(model='openai/gpt-4o', temperature=0.1, top_p=1.0, max_tokens_to_sample=1024, stream=False, functions=None, tools=None, add_generation_kwargs={}, api_base=None) concurrency_limit=16 max_retries=2 initial_backoff=1.0 max_backoff=60.0
2024-08-22 10:06:01,584 - INFO - core.base.providers.llm - Initializing CompletionProvider with config: extra_fields={'concurrent_request_limit': 16} provider='litellm' generation_config=GenerationConfig(model='openai/gpt-4o', temperature=0.1, top_p=1.0, max_tokens_to_sample=1024, stream=False, functions=None, tools=None, add_generation_kwargs={}, api_base=None) concurrent_request_limit=16 max_retries=2 initial_backoff=1.0 max_backoff=60.0
2024-08-22 10:06:02,115 - INFO - core.base.providers.database - Initializing DatabaseProvider with config extra_fields={} provider='postgres'.
2024-08-22 10:06:02,115 - INFO - core.providers.database.vector - Using TCP connection
2024-08-22 10:06:03,320 - INFO - core.providers.database.vector - Successfully initialized PGVectorDB with collection: hyde_cookbook
Expand All @@ -121,13 +121,13 @@ This results in output like this -
INFO: Started server process [42821]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Uvicorn running on http://0.0.0.0:7272 (Press CTRL+C to quit)

```

The script above works by using a custom pipe factory, `R2RPipeFactoryWithMultiSearch` alongside the `R2RBuilder`. The custom factory overrides the default `create_vector_search_pipe` method of `R2RPipeFactory` to create a `MultiSearchPipe`. The `MultiSearchPipe` is a pre-built implementation in R2R which facilitates the conversion of an incoming query into multiple downstream queries. The implementation is actually rather simle, as we can see below:

```python
```python
class R2RPipeFactoryWithMultiSearch(R2RPipeFactory):
...
def create_vector_search_pipe(self, *args, **kwargs):
Expand All @@ -154,7 +154,7 @@ class R2RPipeFactoryWithMultiSearch(R2RPipeFactory):
task_prompt=task_prompt_name,
),
)

# Create search pipe override and pipes
inner_search_pipe = kwargs.get(
"multi_inner_search_pipe_override", None
Expand All @@ -166,7 +166,7 @@ class R2RPipeFactoryWithMultiSearch(R2RPipeFactory):
config=multi_search_config,
)


```


Expand Down Expand Up @@ -273,7 +273,7 @@ if __name__ == "__main__":

### Running Web Search RAG

Now that the server is running we perform search or RAG exactly as above, except now the results will be rooted in web sources.
Now that the server is running we perform search or RAG exactly as above, except now the results will be rooted in web sources.
This will produce output similar to:

```
Expand Down Expand Up @@ -341,4 +341,3 @@ Where to go from here?
1. Experiment with different HyDE prompts to optimize for your specific domain.
2. Try combining HyDE with other R2R features like knowledge graph integration for even more powerful RAG systems.
3. Dive into R2R's documentation to discover more advanced techniques and customization options. For more information, refer [here](/documentation/deep-dive/main/introduction).

Loading
Loading