Installing and Running Ragit
To install and run RAGIT locally the easiest way is to use a virtual machine that can be created using vagrant. You will need to have vagrant installed on your machine .
Assuming you already have vagrant installed then you need to follow these steps to install the repository under your home directory (you can always install it in any other directory if needed).
cd ~
git clone git@github.com:adaptaware/ragit.git
mkdir ~/ragit-data
cd ragit
vagrant up
vagrant ssh
Now you can ssh to the newly created virtual machine which should be ready to go.
The ragit-data directory is shared between the host and guest machine and it contains all the RAG collections.
Each RAG Collection is stored under a subdirectory of the ragit-data directory. The name of the subdirectory is the name of the RAG collection as well. Inside this subdirectory must be another subdirectory called documents which contains all the documents (pdf, markdown, docx etc) that are used for the RAG collection.
The dummy RAG Collection For testing purposes we need one special collection named dummy which will be used by some of the intergration and functional tests for the application.
To create the dummy collection and assuming you are inside the vagrant box follow these steps:
mkdir -p ~/ragit-data/dummy/documents
cp -R /vagrant/ragit/libs/testing_data/* ~/ragit-data/dummy/documents
At this point your ragid-data directory should look similar to the following:
ragit-data/
├── dummy
│ └── documents
│ ├── hello-world.docx
│ ├── hello_world.py
│ ├── nested_dir
│ │ ├── method-chaining.md
│ │ └── nested_2
│ │ └── sample1.pdf
│ ├── patents.pdf
│ ├── sql-alchemy-sucks.md
│ └── sunhaven.md
The required python libraries can be installed as follows:
pip3 install -r /vagrant/requirements.txt
Under your vagrant's machine home directory /home/vagrant create a new file
called settings.json and store into it a valid OpenAI key in the following
format:
{
"OPENAI_API_KEY": "<valid-open-ai-key>",
"VECTOR_DB_PROVIDER": "<supported-vector-db>"
}The supported vector databases are the following:
- CHROMA
- MILVUS
You can verify your setup by running all the tests by following these commands:
cd /vagrant/ragit/
pt
At this point you should have a successfully installed RAGit application along with a RAG collection that you can use similarly to any other RAG collection you are going to create.
Build the backend
To create the necessary backend processing which includes:
- Preprocessing the documents under the RAG collection
- Splitting the documents to chunks
- Creating the embeddings for each chunk
- Insert the updates to the vector database
you should use the ragit command line application which can start by running
the command ragit anywhere from insider the vagrant box which should bring up
the following options:
(ssh)vagrant@ragit:/vagrant/ragit$ ragit
Welcome to the RAG Collection Tracker. Type help or ? to list commands.
(RAGit)
Show all the commands
entering the help or ? we can see all the available commands:
(RAGit) ?
l (list): List all available collections.
s (stats) <name>: Print its stats for the pass in collection.
p (process): Process the data the passed in collection.
i (create_images): Creates the images for the pdfs in the collection.
m (create_markdowns): Creates missing markdowns.
h (help): Prints this help message.
e (exit): Exit.
Show all the RAG collections
to see all the available collections we enter list or l:
(RAGit) list
dummy
(RAGit)
since we only have created only the dummy collection this the only one we
see.
Show all the statistics for a RAG collection
If we press stats dummy or s dummy we see the stats of the statistics for
the dummy collection:
(RAGit) stats dummy
name.....................: dummy
full path................: /home/vagrant/ragit-data/dummy/documents
total documents..........: 5
total documents in db....: 0
total chunks.............: 0
with embeddings..........: 0
without embeddings.......: 0
inserted to vectordb.....: 0
to insert to vector db...: 0
total pdf files..........: 2
pdf missing images.......: 2
images missing markdowns.: 0
(RAGit)
The above result tells us that we have 5 non pdf documents and 2 pdf in the RAG collection.
It also tell us that that the pdf files are missing both their conversion to images and to the corresponding markdowns while nothing is inserted in the database of the vectordb.
Create the images for the pdf
To create the missing images for the pdf we can type the following command i dummy or create_images dummy
(RAGit) create_images dummy
Processing /home/vagrant/ragit-data/dummy/documents/patents.pdf created 2 image files.
Processing /home/vagrant/ragit-data/dummy/documents/nested_dir/nested_2/sample1.pdf created 2 image files.
Created 4 new images.
(RAGit)
Now if we see the statistics of the collecion we are getting the following:
(RAGit) s dummy
name.....................: dummy
full path................: /home/vagrant/ragit-data/dummy/documents
total documents..........: 5
total documents in db....: 0
total chunks.............: 0
with embeddings..........: 0
without embeddings.......: 0
inserted to vectordb.....: 0
to insert to vector db...: 0
total pdf files..........: 2
pdf missing images.......: 0
images missing markdowns.: 4
(RAGit)
Note that now although we have no missing pdf images we are missing the creation of 4 markdowns (those that correctpond to the images we just created).
Create the markdowns
To create the missing markdowns for the dummy collection we need to enter the
following command: m dummy or create_markdowns dummy as can be see here:
(RAGit) create_markdowns dummy
1/4 /home/vagrant/ragit-data/dummy/synthetic/images/documents/nested_dir/nested_2/sample1/sample1_2.png took 5.94 seconds
2/4 /home/vagrant/ragit-data/dummy/synthetic/images/documents/nested_dir/nested_2/sample1/sample1_1.png took 5.16 seconds
3/4 /home/vagrant/ragit-data/dummy/synthetic/images/documents/patents/patents_2.png took 8.63 seconds
4/4 /home/vagrant/ragit-data/dummy/synthetic/images/documents/patents/patents_1.png took 9.00 seconds
(RAGit)
Now the statistics for the dummy collection look as follows:
(RAGit) stats dummy
name.....................: dummy
full path................: /home/vagrant/ragit-data/dummy/documents
total documents..........: 9
total documents in db....: 0
total chunks.............: 0
with embeddings..........: 0
without embeddings.......: 0
inserted to vectordb.....: 0
to insert to vector db...: 0
total pdf files..........: 2
pdf missing images.......: 0
images missing markdowns.: 0
(RAGit)
as we can see we have no more missing images or markdowns. Still, as we can see we need to process the documents and insert them to the vector db which will make the RAG collection ready to serve client queries.
Process the documents
To process the documents we need to enter the process dummy or p dummy
command:
(RAGit) process dummy
Will insert all available chunks to the database.
2024-12-08 02:42:06.446587 5 /home/vagrant/ragit-data/dummy/documents/sunhaven.md
2024-12-08 02:42:06.449392 6 /home/vagrant/ragit-data/dummy/documents/hello_world.py
...
Will insert all available chunks to the database.
2024-12-08 02:42:06.493501 3 /home/vagrant/ragit-data/dummy/synthetic/markdowns/documents/nested_dir/nested_2/sample1/sample1_2.md
2024-12-08 02:42:06.498230 6 /home/vagrant/ragit-data/dummy/synthetic/markdowns/documents/patents/patents_1.md
...
Inserted 49 chunks.
Will insert all available embeddings to the database.
Embeddings count: 1
Embeddings count: 2
Embeddings count: 3
...
Inserted 49 embeddings.
updating the vector db.
1 0
3 1
4 2
....
49 48
Totally inserted records: 49
Inserted 49 chunks to the vector db.
(RAGit)
Now the stats for the collection look as follows:
(RAGit) s dummy
name.....................: dummy
full path................: /home/vagrant/ragit-data/dummy/documents
total documents..........: 9
total documents in db....: 9
total chunks.............: 49
with embeddings..........: 49
without embeddings.......: 0
inserted to vectordb.....: 49
to insert to vector db...: 0
total pdf files..........: 2
pdf missing images.......: 0
images missing markdowns.: 0
(RAGit)
and as we can see we have 49 chunks while all of the were inserted to the vector db and are ready for processing.
Starting RAGit web server
The front end of RAGit is a web-based service that can be started as follows:
(ssh)vagrant@ragit:/vagrant/ragit$ cd /vagrant/ragit/front_end/
(ssh)vagrant@ragit:.../ragit/front_end$ python3 app.py dummy
as we can see we can pass the RAG collection name that we need to process, here
is dummy in the command line; doing so will create some output similar to the
following:
Running the RAGIT UI as not ADMIN
Loading vector db, using collection dummy
2024-12-08 02:51:54,731 - INFO - Loading vector db, using collection dummy
2024-12-08 02:51:54,901 - INFO - Successfully initialized vector db: /home/vagrant/ragit-data/dummy/vectordb/dummy-chroma-vector.db chunk_embeddings gpt-4-turbo
2024-12-08 02:51:55,136 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2024-12-08 02:51:57,548 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-08 02:51:57,557 - INFO - Loading vector db done.. QueryResponse(response='This is about a formal debate on the topic of patents, discussing whether they do more harm than good. The debate took place at the Euroscience Open Forum in Manchester, UK, and featured arguments from both pro- and anti-patent experts. The discussion covered the history of patents, their role in promoting innovation, and the challenges they present, such as high legal costs and accessibility issues for life-saving innovations', temperature=0.2, max_tokens=2000, matches_count=3, prompt='\n Use the following pieces of information enclosed in \n <context> tags to provide an answer to the question \n enclosed in <question> tags.\n <context>\n {context}\n </context>\n <question>\n {question}\n </question>\n ', matches=[('Hello world.\n\n\n\nThis is a test..', 0.23420744471436916, '/home/vagrant/ragit-data/dummy/documents/hello-world.docx', 0), ('Hello world.\n\n\n\nThis is a test..', 0.23420744471436916, '/home/vagrant/ragit-data/dummy/documents/hello-world.docx', 0), ('The Great IP Debate: Do Patents Do More Harm Than Good? # The Great IP Debate: Do Patents Do More Harm Than Good? \n**28 Jul 2016 | Viewpoint Patents** \nA formal debate at Euroscience Open Forum in Manchester pitted pro- and anti-patent experts. You decide the outcome. \n--- \nPatents have been with us since the 17th century. In exchange for disclosing ones invention the state grants a limited legal monopoly over exploitation. In theory the system encourages more innovation for the good of society. But recently voices of dissent have been rising. Legal costs are high. Specialists game the system to their advantage. Life-saving innovations get priced beyond the means of the poor. \nOn July 26 in Manchester UK at Euroscience Open Forum Europes biggest biennial science conference Science|Business Editor-in-Chief Richard L. Hudson organised a formal pro- and anti-patent debate among four experts moderated by Dame Nancy Rothwell President and Vice-Chancellor of the University of Manchester. Herewith his paraphrase of the arguments so you can judge for yourself.', 0.26045281122479924, '/home/vagrant/ragit-data/dummy/synthetic/markdowns/documents/patents/patents_1.md', 'n/a')])
2024-12-08 02:51:57,571 - INFO - Starting ragit on port 13131
======== Running on http://0.0.0.0:13131 ========
(Press CTRL+C to quit)
The query / answer we see here is just a self test to validate the connectivity to the LLM and the vectordb and can differ from run to run.
The server was started in port 13131 while the port to access it from the
'outside' is the one we have specified in the Vagrant file in the following
line:
config.vm.network "forwarded_port", guest: 13131, host: 13132
thus in this case the port will be 13132 (it can be anything else is convinient for the host machine as well).
Accessing the RAGit web page
From the browser we can access the RAGit web page using the following url:
localhost:13132
Sign up / Login The first time we are accessing the RAGit webserver we will need to sign up and then use the user name and password to login to the front end.
Start quering the RAG collection
Once we are signed in, the environment should look familier to other chatbots.
Checking the chunks for the response
If we want to see the documents where the above response was originated from we
click in History option as can be seen here:
Viewing the full document where the chunk is coming from
By clicking the link to the name of the document we can see the full document where the chunk is coming from:
