This is the backend repository for the Soulmate application. It is built with NestJS and provides real-time communication via Socket.IO for video and audio streaming, integrates with Google Cloud Speech-to-Text (STT) and Google Gemini (LLM) for diary generation and emotional analysis, and uses PostgreSQL for data persistence.
- User Authentication: Secure user registration and login (local and social).
- Video & Audio Streaming: Real-time video and audio capture and transmission via Socket.IO.
- Speech-to-Text (STT): Integration with Google Cloud Speech-to-Text for converting audio streams to text.
- Large Language Model (LLM) Integration:
- Real-time diary generation based on user utterances and detected emotions using Google Gemini (gemini-2.5-flash-lite).
- Final diary generation and personalized advice at the end of a session.
- Emotion analysis from video segments.
- Diary Management: Create, update, and retrieve diary entries.
- Emotion Tracking: Store and summarize emotional data during user interactions.
- AWS S3 Integration: For storing video segments.
- Dockerized Environment: Easy setup and deployment using Docker and Docker Compose.
- Redis Integration: For caching and email verification codes.
- Node.js (v18 or higher)
- npm or yarn
- Docker & Docker Compose
- Google Cloud Platform (GCP) Project with:
- Speech-to-Text API enabled
- Service Account Key (JSON format)
- Google Gemini API Key
- AWS S3 Bucket
- AWS API Gateway URL and API Key (for AI video analysis)
Create a .env file in the project root based on .env.example and fill in the required values.
# Database Configuration
DB_HOST=localhost
DB_PORT=5555
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_NAME=soulmate
# JWT Secret (for authentication)
JWT_SECRET=superSecretKey
JWT_SALT=10
JWT_ACCESS_TOKEN_EXPIRATION_TIME=3600 # 1 hour
JWT_REFRESH_TOKEN_EXPIRATION_TIME=604800 # 7 days
# AWS S3 Configuration (for video segments)
AWS_REGION=ap-northeast-2
AWS_S3_BUCKET_NAME=your-s3-bucket-name
AWS_ACCESS_KEY_ID=your-aws-access-key-id
AWS_SECRET_ACCESS_KEY=your-aws-secret-access-key
# Google Cloud STT Configuration
GOOGLE_CLOUD_PROJECT_ID=your-gcp-project-id
GOOGLE_CLOUD_KEYFILE_PATH=/path/to/your/google-cloud-keyfile.json # Absolute path within the Docker container (e.g., /app/responsive-ray-476506-p5-1e58509091ca.json)
# Google Gemini LLM Configuration
GEMINI_API_KEY=your-gemini-api-key
LLM_FLASH_MODEL=gemini-2.5-flash-lite
LLM_DIARY_MODEL=gemini-2.5-flash-lite
LLM_REALTIME_DIARY_MODEL=gemini-2.5-flash-lite
LLM_EMOTION_MODEL=gemini-2.5-flash-lite
LLM_CUSTOM_PROMPT_MODEL=gemini-2.5-flash-lite
LLM_MIN_API_CALL_INTERVAL=6000 # Minimum interval between LLM API calls in milliseconds
# AI API Gateway for video analysis
AI_API_GATEWAY_URL=https://your-api-gateway-url.execute-api.region.amazonaws.com/stage/resource
AI_API_GATEWAY_KEY=your-api-gateway-key
# Email Configuration
EMAIL_SERVICE=gmail # e.g., gmail, outlook, sendgrid
EMAIL_USER=your-email@example.com
EMAIL_PASSWORD=your-email-password
# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
# Video Processing Configuration
VIDEO_SEGMENT_UPLOAD_INTERVAL_MS=10000 # Interval for uploading video segments in milliseconds
VIDEO_FRAME_BUFFER_SIZE=300 # Max frames to buffer for a segmentImportant Note on GOOGLE_CLOUD_KEYFILE_PATH:
When deploying with Docker, the GOOGLE_CLOUD_KEYFILE_PATH in your .env file should point to the absolute path within the Docker container, not your local machine. Ensure your Dockerfile and docker-compose.yml correctly copy and mount this file into the container.
This project uses Docker for containerization.
# Build and run the Docker containers
docker-compose up --build
# To run in detached mode
docker-compose up --build -dThis will start the NestJS application, PostgreSQL database, and Redis.
Ensure you have a PostgreSQL instance running and update your .env file with the correct database credentials.
npm install
# or
yarn installnpm run typeorm migration:run
# or
yarn typeorm migration:runnpm run start:dev
# or
yarn start:devThe application will be accessible at http://localhost:3000 (or your configured port).
Swagger documentation will be available at /api (e.g., http://localhost:3000/api).
To create a new migration:
npm run typeorm migration:create ./src/common/databases/postgre/migrations/YourMigrationName
# or
yarn typeorm migration:create ./src/common/databases/postgre/migrations/YourMigrationNameTo run pending migrations:
npm run typeorm migration:run
# or
yarn typeorm migration:runTo revert the last migration:
npm run typeorm migration:revert
# or
yarn typeorm migration:revert.
├── src/
│ ├── auth/ # Authentication module (JWT, social login, local login)
│ ├── common/ # Common utilities, decorators, DTOs, database configurations, S3, Redis, email, exceptions, pipelines
│ │ ├── databases/
│ │ │ ├── postgre/ # PostgreSQL entities, configuration, transaction manager
│ │ │ ├── redis/ # Redis connection
│ │ │ └── s3/ # S3 video and image upload services
│ │ ├── decorators/ # Custom NestJS decorators
│ │ ├── dto/ # Global DTOs
│ │ ├── email/ # Email sending service
│ │ ├── exceptions/ # Custom exception filters
│ │ ├── filters/ # HTTP exception filters
│ │ └── pipeline/ # Python script execution pipeline
│ ├── diaries/ # Diary management module (CRUD, emotions)
│ ├── emotion/ # Emotion analysis module (controllers, services, repositories)
│ ├── socket/ # Real-time communication via Socket.IO
│ │ ├── llm/ # LLM integration (Gemini service, prompts, types)
│ │ ├── tts/ # (Currently empty) Text-to-Speech integration
│ │ └── video/ # Video & audio streaming, STT, AI processing, BullMQ queue
│ ├── users/ # User management module (CRUD, email verification)
│ ├── app.module.ts # Root module
│ ├── main.ts # Application entry point
│ └── ...
├── test/ # E2E and unit tests
├── docker-compose.yml # Docker Compose setup for services
├── Dockerfile # Dockerfile for the NestJS application
├── package.json # Project dependencies and scripts
├── README.md # Project documentation
└── ...