A comprehensive and efficient email search engine made with Vue.js (Tailwind), Go (Chi), and ZincSearch for storage and retrieval of data.
The email search engine is a useful tool for quickly searching through emails using keywords, with an attractive visual interface. While the use case for this search engine was originally for Enrop Corporation, which no longer operates, it can still be implemented for other purposes if they have a similar email database to Enrop's.
This project consisted of various parts:
- Part 1: Email Database Indexing involved indexing email data for efficient storage and retrieval. This part ensured that the email data was organized and easily accessible. The use of Go (Chi) for the backend and ZincSearch as a database was key to handling large volumes of data with speed and simplicity.
- Part 2: Profiling focused on analyzing the indexed email data to extract key insights and improve the system's performance. This was crucial for understanding usage patterns and identifying processes that consumed excessive time or resources.
- Part 3: Visualizer consisted of developing a visual interface using Vue.js and Tailwind to display the analyzed data interactively, using a table and a details section.
- Part 4: Optimization aimed to enhance the indexing system's performance and efficiency through various optimization techniques, including benchmarking and cleaner coding.
- Part 5: Deployment covered deploying the entire system to a production environment, using Terraform and LocalStack to simulate an AWS cloud deployment.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
Docker and Docker Compose are required to run the project locally. For installation instructions, see the Docker documentation.
You can verify your installation with the following commands:
docker -v
docker compose versionDocker Compose is configured to orchestrate the Backend, Frontend, and ZincSearch engine. Use the following command and wait a few minutes for the containers to be created:
docker compose up --buildThe Enron Mail database is needed to index the data. You can download and decompress this file manually from this link or you can run the provided bash script with the following command in another terminal:
./downloaderDownloading and decompressing the database will take a few minutes.
Once the database is ready, you will need to run the indexing script to upload the data to ZincSearch. Run the indexing script with the name of the database:
./indexer enron_mail_20110402After the script has completed, it will take some time for the Docker container to process all the messages.
You can now open http://localhost:5173/ to access the application locally.
To run the profiling server, add the -prof flag when executing the indexer script. This will start the profiler server in port 6060.
./indexer enron_mail_20110402 -profWhile the profiling server is running, to visualize the graph on the web, please use the following in your terminal:
go tool pprof -http=localhost:8081 http://localhost:6060/debug/pprof/profile\?seconds\=30This will help to visualize the profiling graph and flame graph to revise which processes take more time and how they are running.
You can also save the graph as a PDF file:
go tool pprof -pdf http://localhost:6060/debug/pprof/profile\?seconds\=30PDF examples of the profiling done are saved in the profiling directory of the project.
Additional documentation can be found in the following directories:
- 🖼️ Frontend - frontend directory
- ⚙️ Backend - backend directory
- ⏩ Optimization - profiling directory
- 🚀 Deployment - terraform directory
- Go - Programming Language
- ZincSearch - Search Engine
- Chi - Server Framework
- VueJs - Web Framework
- Tailwind - CSS Framework
- Docker - Containerization Platform
- @avi-2-avi - Software Dev