Skip to content

Latest commit

 

History

History
64 lines (52 loc) · 3.11 KB

README.md

File metadata and controls

64 lines (52 loc) · 3.11 KB

🚀VLM-powered-Multimodal-Conversational-AI-Bot

An advanced AI-powered chatbot that enables users to upload images, ask questions in text or audio, and receive real-time responses in both text and audio formats.

Features

  • Image Upload and Analysis: Accepts uploaded images or captures photos directly via webcam.
  • Speech-to-Text: Converts spoken questions into text using Google SpeechRecognition API.
  • Multimodal Chat Interaction: Integrates visual (image), text, and audio data for comprehensive user interaction.
  • Multi-Round Conversations: Retains conversation context across multiple turns for dynamic and coherent interactions.
  • Real-Time Responses: Provides responses powered by OpenAI's multimodal models.
  • Text-to-Speech: Reads out chatbot responses using pyttsx3 for enhanced accessibility.

Technology Stack

  • Frontend: Streamlit for building an interactive user interface.
  • Backend: OpenAI's multimodal API integration for intelligent conversational processing using state-of-the-art InternVL VLLM model.
  • RunPod: InternVL model's responses are made available by serving and inferring the model using vLLM-powered endpoints provided by RunPod Platform.
  • Cloudinary: For secure image upload and hosting.
  • Google SpeechRecognition API: Converts spoken input to text.
  • Pyttsx3: For generating voice responses.
  • Pillow (PIL): Processes and displays images.

How It Works

  1. Upload or Capture an Image:

    • Upload an image file in .png, .jpg, or .jpeg formats.
    • Alternatively, use your webcam to capture a photo directly.

  2. Ask Your Question:

    • Speak into your microphone or type your question.
    • The bot processes both your question and the image for context.

  3. Receive Responses:

    • Textual responses appear on-screen.
    • Optionally, listen to the response via the text-to-speech feature by pressing the Speak button available below the generated response.

Installation

Prerequisites

  • Python 3.8 or higher.
  • Install dependencies using pip.

Setup

  1. Clone the repository: 
    git clone https://github.com/fork123aniket/VLM-powered-Multimodal-Conversational-AI-Bot.git
cd VLM-powered-Multimodal-Conversational-AI-Bot
  2. Install dependencies: 
    pip install -r requirements.txt
  3. Set up API Keys and Endpoint:
    • Replace placeholders in the script for OpenAI, RunPod Endpoint, and Cloudinary.
  4. Run the application: 
    streamlit run multimodal_chatbot.py

Usage

  1. Launch the Web App: Open the provided local server link in your browser.
  2. Interact: Upload or capture an image, speak, or type your question by pressing the Ask Me Anything! button.
  3. Explore Responses: Read or listen to the chatbot's answers by pressing the Submit button.

Project Structure

├── multimodal_chatbot.py   # Main application script  
├── requirements.txt        # Dependencies  
└── README.md               # Project documentation