[AICUP 2024] Competition-2024-PyTorch-LLMRAG

💬 Applications of RAG and LLM in Financial Q&A

TEAM_6029: Kelvin, Jonathan, Edward, Tom

---

• 玉山人工智慧公開挑戰賽－RAG與LLM在金融問答的應用

在大型語言模型加速催化各式技術的年代，語言模型的開發週期越來越短、效能越來越強。隨著大型語言模型的問世，金融業龐大且複雜的資料已經不再是語料檢索無法高度泛化的障礙，而是逐漸被解決的問題。 本屆挑戰賽聚焦在金融問答領域，提供豐富的資料庫供參賽者使用。參賽者需設計機制以提高檢索結果的準確性，包括從提供的語料中找出完整回答問題的正確資料等基本要求，以及應用大型語言模型的生成能力，產出正確且完整的回答。

📌 Quick Inference

To reproduce our submit inference results, please following instructions.

Step 0: Environment Setting

• Download the Repo

  git clone https://github.com/FanChiMao/Competition-2024-PyTorch-LLMRAG.git
  cd Competition-2024-PyTorch-LLMRAG
  git submodule update --init
  

• Prepare the environment
  ❗ Noted: Please check your GPU and OS environment, and go to the PyTorch Website to install Pytorch first.

  conda create --name LLMRAG python=3.10  # to reproduce the results, you have to install python 3.10
  pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118  # take cuda 11.8 as example
  pip install -r requirements.txt
  

Step 1: Preparing Datasets

• Go to the official website to download the datasets. (due to the policy, we can't provide the dataset)

• You can directly run the script

  cd scripts
  1.download_preliminary_data.bat
  

  or run the snippet at ./datasets/download_preliminary_datasets.py

  cd datasets
  python ./download_preliminary_datasets.py
  

• Place the dataset in ./datasets.

Step 2: Running Baseline

• You can directly run the script to run the baseline code

  cd scripts
  2.run_baseline_code.bat
  

  or run the snippet at ./main_baseline.py

  python ./main_baseline.py
  

• After running the baseline code, it will generate the json result on ./output/baseline.json

Step 3: Reproduce Results

• To reproduce our submitted results, you can run

  cd scripts
  3.run_preliminary_results.bat
  

  or run the snippet at ./main_preliminary.py

  python ./preliminary_results.py
  

• After running the baseline code, it will generate the json result on ./output/preliminary_results.json

🕵️ Evaluation

To evaluate the precision@1 for the output json, please following the command

python ./evaluation.py --gt [path of ground_truths_example.json] --rs [path of output json]

take baseline result for example:

python ./evaluation.py --gt ./datasets/preliminary/ground_truths_example.json --rs ./outputs/baseline.json

📫 Contact Us

• Kelvin: fxp61005@gmail.com
• Jonathan: qaz5517359@gmail.com
• Edward:
• Tom: