BestCity DevOps Implementation

📋 Overview

This repository contains the complete DevOps implementation for the BestCity real estate investment platform. It includes Docker containerization, Infrastructure as Code (Terraform), logging & monitoring setup, and AWS automation scripts.

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    Docker Compose Stack                  │
├─────────────────────────────────────────────────────────┤
│                                                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │   BestCity   │  │   MongoDB    │  │   Fluentd    │  │
│  │  Application │  │   Database   │  │   Logging    │  │
│  │  (Port 3099) │  │ (Port 27017) │  │ (Port 24224) │  │
│  └──────────────┘  └──────────────┘  └──────────────┘  │
│                                                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │  Prometheus  │  │   Grafana    │  │     Node     │  │
│  │  Monitoring  │  │ Dashboards   │  │   Exporter   │  │
│  │  (Port 9090) │  │ (Port 3000)  │  │ (Port 9100)  │  │
│  └──────────────┘  └──────────────┘  └──────────────┘  │
│                                                           │
└─────────────────────────────────────────────────────────┘

📁 Repository Structure

bc-ops/
├── Dockerfile                    # Multi-stage Docker build for BestCity app
├── docker-compose.yml            # Complete stack orchestration
├── .env.example                  # Environment variables template
│
├── terraform/                    # Infrastructure as Code
│   ├── main.tf                   # Main Terraform configuration
│   ├── variables.tf              # Variable definitions
│   └── terraform.tfvars.example  # Example values
│
├── fluentd/                      # Logging configuration
│   ├── Dockerfile                # Custom Fluentd image
│   └── conf/
│       └── fluent.conf           # Log aggregation rules
│
├── prometheus/                   # Monitoring configuration
│   └── prometheus.yml            # Metrics collection config
│
├── grafana/                      # Visualization setup
│   └── provisioning/
│       └── datasources/
│           └── datasource.yml    # Prometheus datasource
│
└── scripts/                      # Automation scripts
    ├── setup.sh                  # Local setup script
    ├── deploy.sh                 # EC2 deployment script
    ├── run-local.sh              # Quick local run
    └── aws-cli-tasks.sh          # AWS operations script

🚀 Quick Start

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+
• Node.js 18+ (for local development)
• AWS CLI 2.x (for cloud deployment)
• Terraform 1.0+ (for infrastructure provisioning)

Local Setup (Docker)

1. Clone the repository and navigate to bc-ops:

   cd bc-ops

2. Setup environment:

   cp .env.example .env
   # Edit .env with your configuration

3. Run the setup script:

   ./scripts/setup.sh

   This will:

   • Build Docker images
   • Start all services
   • Run health checks
   • Display service URLs

4. Access the application:

   • Application: http://localhost:3099
   • Prometheus: http://localhost:9090
   • Grafana: http://localhost:3000 (admin/admin)

Local Development (Without Docker)

./scripts/run-local.sh

This runs the app in development mode with hot reload.

🐳 Docker Implementation

Dockerfile Features

• Multi-stage build for optimized image size
• Non-root user for security
• Health checks for container monitoring
• Tini init system for proper signal handling
• Production-ready with minimal attack surface

Docker Compose Services

1. app - BestCity application (React + Node.js)
2. mongodb - Database with persistence
3. fluentd - Centralized logging
4. prometheus - Metrics collection
5. grafana - Metrics visualization
6. node-exporter - System metrics

Managing Services

# Start services
docker-compose up -d

# View logs
docker-compose logs -f [service-name]

# Stop services
docker-compose down

# Stop and remove volumes
docker-compose down -v

# Rebuild specific service
docker-compose up -d --build app

# Scale service (if applicable)
docker-compose up -d --scale app=3

☁️ Cloud Deployment (AWS)

Infrastructure Provisioning with Terraform

1. Navigate to terraform directory:

   cd terraform

2. Configure your variables:

   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your values

3. Initialize Terraform:

   terraform init

4. Plan the infrastructure:

   terraform plan

5. Apply the configuration:

   terraform apply

   This creates:

   • VPC with public subnet
   • Internet Gateway
   • Security Groups
   • EC2 instance (t3.medium)
   • Elastic IP
   • IAM roles and policies

6. Get outputs:

   terraform output

Deploying Application to EC2

After provisioning infrastructure:

# Set environment variables
export EC2_HOST=$(terraform output -raw instance_public_ip)
export SSH_KEY=/path/to/your/key.pem

# Run deployment script
./scripts/deploy.sh

Manual Deployment Steps

If you prefer manual deployment:

1. SSH to EC2:

   ssh -i your-key.pem ec2-user@<EC2_IP>

2. Clone the repository:

   git clone <your-repo-url>
   cd bc-ops

3. Configure environment:

   cp .env.example .env
   nano .env  # Update with your values

4. Start services:

   docker-compose up -d

📊 Monitoring & Logging

Prometheus Metrics

Access Prometheus at http://<host>:9090

Available metrics:

• System metrics (CPU, Memory, Disk, Network)
• Container metrics
• Application metrics (if implemented)

Useful queries:

# CPU usage
rate(node_cpu_seconds_total[5m])

# Memory usage
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Container CPU
rate(container_cpu_usage_seconds_total[5m])

Grafana Dashboards

Access Grafana at http://<host>:3000

• Default credentials: admin/admin
• Prometheus datasource is pre-configured
• Import community dashboards for Node Exporter

Fluentd Logging

Logs are collected from all Docker containers and:

• Stored in /fluentd/log directory
• Tagged by service
• Formatted as JSON
• Rotated daily with compression

View logs:

docker-compose logs -f fluentd

🔧 AWS CLI Operations

The aws-cli-tasks.sh script provides interactive AWS operations:

./scripts/aws-cli-tasks.sh

Features:

• S3 bucket management
• File upload/download
• Application backups
• EC2 instance management
• CloudWatch alarms
• Secrets Manager integration

Example: Create backup

# Set environment
export AWS_REGION=us-east-1
export PROJECT_NAME=bestcity
export ENVIRONMENT=dev

# Run script
./scripts/aws-cli-tasks.sh
# Select option 5 for application backup

🔐 Security Considerations

Docker Security

• ✅ Non-root user in containers
• ✅ Read-only root filesystem (where applicable)
• ✅ No privileged containers
• ✅ Health checks enabled
• ✅ Resource limits set

AWS Security

• ✅ Security groups with minimal ports
• ✅ IAM roles with least privilege
• ✅ Encrypted EBS volumes
• ✅ VPC isolation
• ✅ SSH key-based authentication

Application Security

• ✅ Environment variables for secrets
• ✅ HTTPS ready (add certificate)
• ✅ MongoDB authentication enabled
• ✅ CORS configured
• ✅ Input validation

⚠️ Important: Before production:

1. Change default passwords (Grafana, MongoDB)
2. Add SSL/TLS certificates
3. Restrict SSH access to specific IPs
4. Enable AWS CloudTrail
5. Set up automated backups
6. Configure proper secret management

📝 Environment Variables

Key environment variables (see .env.example):

# Application
NODE_ENV=production
PORT=3099

# Database
MONGO_URI=mongodb://mongodb:27017/bestcity

# Cloudinary (required)
CLOUDINARY_NAME=your_cloudinary_name
CLOUDINARY_API_KEY=your_api_key
CLOUDINARY_API_SECRET=your_api_secret

# JWT
JWT_SECRET=your_secret_key
JWT_EXPIRE=7d

# Email
SENDGRID_API_KEY=your_sendgrid_key

🧪 Testing

Test Docker Build

docker build -f Dockerfile -t bestcity:test ../demo-version
docker run -p 3099:3099 bestcity:test

Test Health Endpoint

curl http://localhost:3099/api/health

Load Testing

# Install Apache Bench
sudo apt-get install apache2-utils  # Ubuntu/Debian
brew install ab  # macOS

# Run load test
ab -n 1000 -c 10 http://localhost:3099/

📚 Additional Resources

Terraform Resources Created

• aws_vpc.main - Virtual Private Cloud
• aws_subnet.public - Public subnet
• aws_internet_gateway.main - Internet gateway
• aws_security_group.app_sg - Security group
• aws_instance.app_server - EC2 instance
• aws_eip.app_eip - Elastic IP
• aws_iam_role.ec2_role - IAM role

Useful Commands

# Check Docker resource usage
docker stats

# Cleanup Docker system
docker system prune -a

# Backup MongoDB
docker exec bestcity-mongodb mongodump --out=/backup

# Restore MongoDB
docker exec bestcity-mongodb mongorestore /backup

# View Terraform state
terraform show

# Destroy infrastructure
terraform destroy

🐛 Troubleshooting

Application won't start

1. Check logs:

   docker-compose logs app

2. Verify environment variables:

   docker-compose config

3. Check MongoDB connection:

   docker-compose exec mongodb mongosh --eval "db.stats()"

Port conflicts

# Check what's using the port
lsof -i :3099

# Change ports in docker-compose.yml

Terraform errors

# Validate configuration
terraform validate

# Check AWS credentials
aws sts get-caller-identity

# Enable debug logging
export TF_LOG=DEBUG
terraform apply

🤝 Contributing

This is a test project demonstrating DevOps practices. Key areas covered:

1. ✅ Containerization - Multi-stage Docker build
2. ✅ Orchestration - Docker Compose with multiple services
3. ✅ Infrastructure as Code - Terraform for AWS
4. ✅ Monitoring - Prometheus + Grafana stack
5. ✅ Logging - Centralized with Fluentd
6. ✅ Automation - Shell scripts for common tasks
7. ✅ AWS Integration - CLI operations and cloud deployment
8. ✅ Security - Best practices implemented
9. ✅ Documentation - Comprehensive guides

📄 License

This is a test project for DevOps evaluation purposes.

📧 Support

For issues or questions related to this DevOps implementation, please:

1. Check the troubleshooting section
2. Review the logs
3. Verify your configuration matches the examples

---

Built with ❤️ for DevOps Excellence