Feature/add document summary to ingestion

[emrgnt-cmplxty]

Important

Adds document summary generation to ingestion process and refactors search settings to SearchSettings.

• Behavior:
  • Introduces document summary generation during ingestion in ingestion_service.py and ingestion_workflow.py.
  • Adds augment_document_info() to generate summaries using LLM and store embeddings.
  • Updates ingestion status to include AUGMENTING.
• Search Settings:
  • Renames VectorSearchSettings and DocumentSearchSettings to SearchSettings across multiple files.
  • Updates search methods to use SearchSettings in retrieval_service.py, retrieval_router.py, and vector_search_pipe.py.
• Database:
  • Modifies PostgresDocumentHandler to include summary and summary_embedding fields.
  • Adds full-text and semantic search capabilities in document.py.
• Configuration:
  • Updates configuration files to include document summary settings.
  • Adds default_summary.yaml for summary prompt configuration.
• Misc:
  • Refactors search pipelines and pipes to accommodate new search settings.
  • Updates API models and responses to reflect changes in document search results.

^{This description was created by for 39c5fae. It will automatically update as commits are pushed.}

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
yc-demo	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 11, 2024 11:24pm

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 39c5fae in 1 minute and 17 seconds

More details

• Looked at 1959 lines of code in 32 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. py/shared/abstractions/search.py:184

• Draft comment:
  The SearchSettings class combines functionalities of both VectorSearchSettings and DocumentSearchSettings. Ensure that this change is well-documented to avoid confusion. Also, consider removing or refactoring the filters field since it's marked as deprecated but still used in the code.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The PR involves renaming VectorSearchSettings and DocumentSearchSettings to SearchSettings. This change is consistent across the codebase, but there are some potential issues with backward compatibility and clarity. The new SearchSettings class combines functionalities of both previous classes, which might lead to confusion if not documented properly. Additionally, the filters field is marked as deprecated, but it's still being used in the code. This could lead to confusion for developers using this codebase.

Workflow ID: wflow_yjRDsMIvzCXNEsvb

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.