Compare RAGAs metrics

samples/mean_metric_ensemble.csv

@@ -0,0 +1,5 @@
+,0
+faithfulness,0.8
+answer_relevancy,0.7303519445666862
+context_precision,0.7778333333140975
+context_recall,0.95

samples/mean_metrics.csv

@@ -0,0 +1,5 @@
+,0
+faithfulness,0.7333333333333333
+answer_relevancy,0.7359878053275298
+context_precision,0.8733333333151666
+context_recall,1.0

samples/ragas.csv

@@ -0,0 +1,38 @@
+,question,answer,contexts,ground_truth,context_precision,context_recall,faithfulness,answer_relevancy
+0,What should be included in the invoice?,"The invoice should include the following items:
+
+1. The Contract Number
+2. Contractor’s name and address
+3. A description of the Services performed
+4. The dates such Services were performed
+5. A pricing calculation based on the payment terms
+6. Properly reimbursable expenses (if any) incurred along with receipts for such expenses for all individual expenses exceeding $25 USD
+7. Such other information as Mercy Corps may reasonably request.","['Invoicing and Payment.'
+ 'have no obligation to pay any portion of invoices received more than 60 days after the end date of the Contract. Each invoice will include (i)\xa0the Contract Number; (ii)\xa0Contractor’s name and address; (iii)\xa0a description of the Services performed, (iv)\xa0the dates such Services were performed, (v)\xa0a pricing calculation based on the payment terms, (vi)\xa0properly reimbursable expenses (if any) incurred along with receipts for such expenses (if applicable) for all individual expenses exceeding $25'
+ 'for all individual expenses exceeding $25 USD, and (vii) such other information as Mercy Corps may reasonably request.  Invoices will only be deemed received on the date they are delivered to the Authorized Representative pursuant to the Payment Terms (see Schedule I).  If Mercy Corps determines that Services that are the subject of an invoice have not been performed in accordance with the Statement of Services, Mercy Corps may dispute the invoice by sending Contractor notice of such dispute'
+ 'Terms, and the time required of Contractor to complete the Services.'
+ 'Liability/Indemnity\nUnder no circumstances nor for any reason whatsoever will the European Union be held liable for damages as a result of the work pursuant to this Contract.  \nRight of Access/ Audit\nThe Vendor will be responsible for holding all invoices, receipts and financial and accounting documents relating to this Contract for at least seven years following final payment made under this Contract.'
+ 'Except as otherwise provided in the Statement of Services, Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) in accordance with the Payment Terms within 30 days after the later of (i)\xa0receipt of the invoice or (ii)\xa0resolution of the items set forth in the notice of disputed charges.\nMercy Corps may off-set any amount it owes Contractor against any amount Contractor owes Mercy Corps.\nTaxes, Duties and Expenses.']","Each invoice will include (i) the Contract Number; (ii) Contractor’s name and address; (iii) a description of the Services performed, (iv) the dates such Services were performed, (v) a pricing calculation based on the payment terms, (vi) properly reimbursable expenses (if any) incurred along with receipts for such expenses (if applicable) for all individual expenses exceeding $25 USD, and (vii) such other information as Mercy Corps may reasonably request.",0.9999999999833333,1.0,1.0,0.9815858311463699
+1,What happens if a contractor breaches the contract?,"If a contractor breaches the contract, Mercy Corps may terminate the contract immediately upon written notice. Mercy Corps may withhold any amounts owed to the contractor until the breach is remedied. Additionally, if the termination is due to the contractor's breach, Mercy Corps will not be obligated to pay the contractor for any partially completed work.","['Terms, and the time required of Contractor to complete the Services.'
+ 'Contractor possesses all governmental and other certifications and licenses necessary to perform the Services.  Performance by Contractor of its obligations under this Contract will not infringe on any patent, copyright, trademark, trade secret or other proprietary right of any third party.\nContractor will comply with all applicable law, regulations and rules in the performance of its obligations under this Contract.'
+ 'by Mercy Corps immediately upon written notice if Mercy Corps using its sole discretion determines that Contractor has or will breach any of its warranties, covenants or representations in this Contract, in which case Mercy Corps may withhold any and all amounts owed to Contractor until such breach is remedied.'
+ 'Contractor has not engaged in, and will not engage in, any of the following conduct:  (A) trafficking in persons (as defined in the Protocol to Prevent, Suppress, and Punish Trafficking in Persons, especially Women and Children, supplementing the UN Convention against Transnational Organized Crime); (B) procuring a commercial sex act; or (C) using forced labor.'
+ 'Definitions:' 'Invoicing and Payment.' 'Anti-terrorism'
+ 'of business on the next business day for the recipient).'
+ 'and also may require the Contractor to refund a portion or all of any amounts paid to the Contractor under this contract.'
+ 'In the event of termination due to Contractor’s breach of this Contract or by Contractor for Contractor’s convenience, Mercy Corps will not be obligated to pay Contractor for any partially completed work.  In the event termination is due to Mercy Corps’ breach of this Contract, by Mercy Corps for Mercy Corps’ convenience, due to force majeure event, or due to loss of funding, Mercy Corps will be obligated to pay Contractor for its reasonable, pro-rated costs of work completed and expenses']","In the event of termination due to Contractor’s breach of this Contract or by Contractor for Contractor’s convenience, Mercy Corps will not be obligated to pay Contractor for any partially completed work. ",0.344444444432963,1.0,1.0,0.8543245266783456
+2,What is the deadline for submitting the invoicing schedule?,I don't know.,"['Invoicing and Payment.'
+ 'Contractor will submit invoices to Mercy Corps in accordance with the invoicing schedule and invoicing delivery terms set forth in the Statement of Services (Schedule I). Final invoices must be submitted within 60 days of the end date of the Contract. Contractor recognizes that in many cases Mercy Corps’ donor will not reimburse Mercy Corps for invoices submitted beyond 60 days after the termination of a contract and therefore Mercy Corps will have no obligation to pay any portion of invoices'
+ 'have no obligation to pay any portion of invoices received more than 60 days after the end date of the Contract. Each invoice will include (i)\xa0the Contract Number; (ii)\xa0Contractor’s name and address; (iii)\xa0a description of the Services performed, (iv)\xa0the dates such Services were performed, (v)\xa0a pricing calculation based on the payment terms, (vi)\xa0properly reimbursable expenses (if any) incurred along with receipts for such expenses (if applicable) for all individual expenses exceeding $25'
+ 'for all individual expenses exceeding $25 USD, and (vii) such other information as Mercy Corps may reasonably request.  Invoices will only be deemed received on the date they are delivered to the Authorized Representative pursuant to the Payment Terms (see Schedule I).  If Mercy Corps determines that Services that are the subject of an invoice have not been performed in accordance with the Statement of Services, Mercy Corps may dispute the invoice by sending Contractor notice of such dispute']",Final invoices must be submitted within 60 days of the end date of the Contract. Contractor recognizes that in many cases Mercy Corps’ donor will not reimburse Mercy Corps for invoices submitted beyond 60 days after the termination of a contract and therefore Mercy Corps will have no obligation to pay any portion of invoices received more than 60 days after the end date of the Contract.,0.999999999975,1.0,0.0,0.0
+3,When will Mercy Corps submit payment?,Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) within 30 days after the later of (i) receipt of the invoice or (ii) resolution of the items set forth in the notice of disputed charges.,"['Contractor will submit invoices to Mercy Corps in accordance with the invoicing schedule and invoicing delivery terms set forth in the Statement of Services (Schedule I). Final invoices must be submitted within 60 days of the end date of the Contract. Contractor recognizes that in many cases Mercy Corps’ donor will not reimburse Mercy Corps for invoices submitted beyond 60 days after the termination of a contract and therefore Mercy Corps will have no obligation to pay any portion of invoices'
+ 'Except as otherwise provided in the Statement of Services, Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) in accordance with the Payment Terms within 30 days after the later of (i)\xa0receipt of the invoice or (ii)\xa0resolution of the items set forth in the notice of disputed charges.\nMercy Corps may off-set any amount it owes Contractor against any amount Contractor owes Mercy Corps.\nTaxes, Duties and Expenses.'
+ 'Delivery of Services.   \nContractor will perform the Services, and Mercy Corps will pay for the Services, in accordance with the terms and conditions and within the Performance Period set forth in this Contract and the Statement of Services.'
+ 'for all individual expenses exceeding $25 USD, and (vii) such other information as Mercy Corps may reasonably request.  Invoices will only be deemed received on the date they are delivered to the Authorized Representative pursuant to the Payment Terms (see Schedule I).  If Mercy Corps determines that Services that are the subject of an invoice have not been performed in accordance with the Statement of Services, Mercy Corps may dispute the invoice by sending Contractor notice of such dispute'
+ 'by Mercy Corps immediately upon written notice if Mercy Corps using its sole discretion determines that Contractor has or will breach any of its warranties, covenants or representations in this Contract, in which case Mercy Corps may withhold any and all amounts owed to Contractor until such breach is remedied.']","Except as otherwise provided in the Statement of Services, Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) in accordance with the Payment Terms within 30 days after the later of (i) receipt of the invoice or (ii) resolution of the items set forth in the notice of disputed charges.",0.99999999998,1.0,1.0,0.9458873108716661
+4,Who is responsible for paying taxes in the agreement?,"The Contractor is responsible for all expenses incurred in performing under the Contract, including all taxes, duties, and other governmental charges. However, if the law requires Mercy Corps to withhold taxes from payments to the Contractor, Mercy Corps may withhold those taxes and pay them to the appropriate taxing authority.","['Except as otherwise provided in the Statement of Services, Contractor is responsible for all expenses incurred by it in performing under this Contract and all taxes, duties and other governmental charges with respect to the provision of Services.  If the law requires Mercy Corps to withhold taxes from payments to Contractor, Mercy Corps may withhold those taxes and pay them to the appropriate taxing authority.  Mercy Corps will deliver to Contractor an official notice for such taxes.  Mercy'
+ 'Invoicing and Payment.'
+ 'paid (at each of their own cost, unless otherwise agreed).'
+ 'Liability/Indemnity\nUnder no circumstances nor for any reason whatsoever will the European Union be held liable for damages as a result of the work pursuant to this Contract.  \nRight of Access/ Audit\nThe Vendor will be responsible for holding all invoices, receipts and financial and accounting documents relating to this Contract for at least seven years following final payment made under this Contract.'
+ 'The Vendor will seek to ensure that none of the funds or assets provided under this Agreement are made available or used to provide support to individuals, groups or entities associated with terrorism.'
+ 'Except as otherwise provided in the Statement of Services, Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) in accordance with the Payment Terms within 30 days after the later of (i)\xa0receipt of the invoice or (ii)\xa0resolution of the items set forth in the notice of disputed charges.\nMercy Corps may off-set any amount it owes Contractor against any amount Contractor owes Mercy Corps.\nTaxes, Duties and Expenses.']","Except as otherwise provided in the Statement of Services, Contractor is responsible for all expenses incurred by it in performing under this Contract and all taxes, duties and other governmental charges with respect to the provision of Services.  If the law requires Mercy Corps to withhold taxes from payments to Contractor, Mercy Corps may withhold those taxes and pay them to the appropriate taxing authority.  Mercy Corps will deliver to Contractor an official notice for such taxes.  Mercy Corps will use reasonable efforts to minimize any taxes withheld to the extent allowed by law.",0.9666666666473333,0.75,1.0,0.8967428990913818

scripts/evaluate.py

@@ -11,61 +11,92 @@
 import seaborn as sns
 import matplotlib.pyplot as plt
 from matplotlib.colors import LinearSegmentedColormap
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.docstore.document import Document
+import fitz  # PyMuPDF
+from docx import Document as DocxDocument
+from langchain.retrievers import EnsembleRetriever
+from langchain_community.retrievers import BM25Retriever
+
 
 load_dotenv()
 
 llm = ChatOpenAI(
-        model_name="gpt-3.5-turbo",
+        model_name="gpt-4o-mini",
         api_key=os.getenv("OPENAI_API_KEY"),
         temperature=0,
         max_tokens=800,
         model_kwargs={"top_p": 0, "frequency_penalty": 0, "presence_penalty": 0},
     )
-# Path to the Chroma DB
-chroma_db_path = "/home/daisy/Desktop/tenx/ContractAdvisorRAG/scripts/db1"
-
-# Load Chroma DB vector store
-vectorstore = Chroma(
-    persist_directory=chroma_db_path,
-    embedding_function=OpenAIEmbeddings()
-)
-
-# Initialize retriever from the vector store
-#retriever = vectorstore.as_retriever()
-
-#multiquery retriever
+#text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
+embedding = OpenAIEmbeddings()
+
+def extract_text_from_docx(file_path):
+    text = ""
+    doc = DocxDocument(file_path)
+    for para in doc.paragraphs:
+        text += para.text + "\n"
+    return text
+def extract_text_from_pdf(file_path):
+    text = ""
+    with fitz.open(file_path) as doc:
+        for page in doc:
+            text += page.get_text()
+    return text
+
+file_paths = ["/home/daisy/Desktop/tenx/ContractAdvisorRAG/samples/Sample_Contract.docx"]
+texts = []
+
+for file_path in file_paths:
+    if file_path.endswith('.pdf'):
+        texts.append(extract_text_from_pdf(file_path))
+    elif file_path.endswith('.docx'):
+        texts.append(extract_text_from_docx(file_path))
+
+
+chunk_size = 500
+chunk_overlap = 50
+text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
+
+chunks = []
+for text in texts:
+    chunks.extend(text_splitter.split_text(text))
+
+documents = [Document(page_content=chunk) for chunk in chunks]
+
+# bm25_retriever = BM25Retriever.from_documents(documents)
+# bm25_retriever.k = 2
+
+# chroma_vectorstore = Chroma.from_documents(documents, embedding)
+# chroma_retriever = chroma_vectorstore.as_retriever()
+
+# ensemble_retriever = EnsembleRetriever(
+#     retrievers=[bm25_retriever, chroma_retriever], weights=[0.5, 0.5]
+# )
+
+# multiquery
+vectorstore = Chroma.from_documents(documents=documents, embedding=embedding)
 retriever = MultiQueryRetriever.from_llm(
-    retriever=vectorstore.as_retriever(), llm=llm
-)
-# Set up the RAG chain
-rag_chain = RetrievalQA.from_chain_type(llm, chain_type="stuff", retriever=retriever)
-
-def rag_qa(query):
-    response = rag_chain.invoke(query)
-    return response
+    retriever=vectorstore.as_retriever(), llm=llm)
 
-#query = "In which street does the Advisor live?"
-#answer = rag_qa(query)
-#print(answer)
+rag_chain = RetrievalQA.from_chain_type(llm, chain_type="stuff", retriever=retriever)
 
 #RAGAS
 from datasets import Dataset
 
 questions = [
-    "Who are the parties to the Agreement and what are their defined names?", 
-    "What is the termination notice?",
-    "In which street does the Advisor live?",
-    "How much is the escrow amount?",
-    "Does any of the Sellers provide a representation with respect to any Tax matters related to the Company?",
-    "Whose consent is required for the assignment of the Agreement by the Buyer?",
+    "What should be included in the invoice?", 
+    "What happens if a contractor breaches the contract?",
+    "What is the deadline for submitting the invoicing schedule?",
+    "When will Mercy Corps submit payment?",
+    "Who is responsible for paying taxes in the agreement?"
 ]
 ground_truths = [
-    "Cloud Investments Ltd. (“Company”) and Jack Robinson (“Advisor”)",
-    "According to section 4:14 days for convenience by both parties. The Company may terminate without notice if the Advisor refuses or cannot perform the Services or is in breach of any provision of this Agreement.",
-    "1 Rabin st, Tel Aviv, Israel",
-    "The escrow amount is equal to $1,000,000.",
-    "No. Only the Company provides such a representation. ",
-    "If the assignment is to an Affiliate or purchaser of all of the Buyer assets, no consent is required. Otherwise, the consent of the Company and the Seller Representative is required."
+    "Each invoice will include (i) the Contract Number; (ii) Contractor’s name and address; (iii) a description of the Services performed, (iv) the dates such Services were performed, (v) a pricing calculation based on the payment terms, (vi) properly reimbursable expenses (if any) incurred along with receipts for such expenses (if applicable) for all individual expenses exceeding $25 USD, and (vii) such other information as Mercy Corps may reasonably request.",
+    "In the event of termination due to Contractor’s breach of this Contract or by Contractor for Contractor’s convenience, Mercy Corps will not be obligated to pay Contractor for any partially completed work. ",
+    "Final invoices must be submitted within 60 days of the end date of the Contract. Contractor recognizes that in many cases Mercy Corps’ donor will not reimburse Mercy Corps for invoices submitted beyond 60 days after the termination of a contract and therefore Mercy Corps will have no obligation to pay any portion of invoices received more than 60 days after the end date of the Contract.",
+    "Except as otherwise provided in the Statement of Services, Mercy Corps will pay each invoice (or adjusted invoice if the subject of dispute) in accordance with the Payment Terms within 30 days after the later of (i) receipt of the invoice or (ii) resolution of the items set forth in the notice of disputed charges.",
+    "Except as otherwise provided in the Statement of Services, Contractor is responsible for all expenses incurred by it in performing under this Contract and all taxes, duties and other governmental charges with respect to the provision of Services.  If the law requires Mercy Corps to withhold taxes from payments to Contractor, Mercy Corps may withhold those taxes and pay them to the appropriate taxing authority.  Mercy Corps will deliver to Contractor an official notice for such taxes.  Mercy Corps will use reasonable efforts to minimize any taxes withheld to the extent allowed by law."
 ]
 
 answers = []
@@ -111,8 +142,9 @@ def rag_qa(query):
 )
 
 df = result.to_pandas()
-
-#print(df.head())
+print(df.head())
+mean_df=df[['faithfulness','answer_relevancy', 'context_precision', 'context_recall']].mean(axis=0)
+mean_df.to_csv('mean_metric_ensemble.csv, encoding="utf-8", index=True')
 heatmap_data = df[['context_precision', 'context_recall', 'faithfulness', 'answer_relevancy']]
 
 cmap = LinearSegmentedColormap.from_list('green_red', ['red', 'green'])