RabbitMQ Python Tutorial

A simple demonstration of message queuing with RabbitMQ, Docker, and Python. This project shows how to send and receive messages using a producer-consumer pattern with containerized services.

What This Does

This project implements a Work Queue (Task Queue) pattern using RabbitMQ - a messaging system that distributes time-consuming tasks among multiple workers. This is ideal for background job processing where you want to avoid doing resource-intensive tasks immediately and waiting for them to complete.

Implementation Type: Work Queue with Reliability Features

This implementation includes production-ready reliability features:

• ✅ Durable Queues: Queue survives RabbitMQ broker restarts
• ✅ Persistent Messages: Messages survive broker restarts
• ✅ Manual Acknowledgments: Messages aren't lost if a worker fails during processing
• ✅ Fair Dispatch: Workers receive only one message at a time (prevents overload)

Components

• Producer: Sends task messages to a RabbitMQ queue
• Consumer: Receives and processes tasks from the queue
• RabbitMQ: Message broker that manages the queue and ensures delivery
• Docker: Everything runs in containers for easy setup

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│  🐳 DOCKER ENVIRONMENT                                                       │
│                                                                             │
│     ┌───────────────────┐                 ┌───────────────────┐             │
│     │     PRODUCER      │                 │     CONSUMER      │             │
│     │    (Container)    │                 │    (Container)    │             │
│     ├───────────────────┤                 ├───────────────────┤             │
│     │ script:           │                 │ script:           │             │
│     │  producer.py      │                 │  consumer.py      │             │
│     │                   │                 │                   │             │
│     │ action:           │                 │ action:           │             │
│     │  • Sends msg      │                 │  • Receives msg   │             │
│     │  • Exits          │                 │  • Processes      │             │
│     └─────────┬─────────┘                 └─────────▲─────────┘             │
│               │                                     │                       │
│               │ (AMQP)                              │ (AMQP)                │
│               │ "Hello RabbitMQ!"                   │                       │
│               │                                     │                       │
│     ┌─────────▼─────────────────────────────────────┴─────────┐             │
│     │                  RABBITMQ CONTAINER                     │             │
│     │               Image: rabbitmq:3-management              │             │
│     │                                                         │             │
│     │   ┌─────────────────────────────────────────────────┐   │             │
│     │   │ ≡ QUEUE: task_queue                             │   │             │
│     │   │   (durable, persistent)                         │   │             │
│     │   └─────────────────────────────────────────────────┘   │             │
│     │                                                         │             │
│     │   INTERNAL PORTS:                                       │             │
│     │   • 5672  (AMQP Messaging)                              │             │
│     │   • 15672 (Management UI)                               │             │
│     └──────────────────────────┬──────────────────────────────┘             │
│                                │                                            │
│  ══════════════════════════════╪══════════════════════════════════          │
│         NETWORK BRIDGE:        │ rabbitmq-net                               │
│  ══════════════════════════════╪══════════════════════════════════          │
│                                │                                            │
└────────────────────────────────┼────────────────────────────────────────────┘
                                 │
                                 ▼
                     ┌───────────────────────┐
                     │     HOST MACHINE      │
                     └───────────┬───────────┘
                                 │
                  ┌──────────────┴──────────────┐
                  │   Exposed Ports (Mapped)    │
                  ├─────────────────────────────┤
                  │ 1. App Access (AMQP)        │
                  │    localhost:5672           │
                  │                             │
                  │ 2. Management Dashboard     │
                  │    http://localhost:15672   │
                  └─────────────────────────────┘
                  
                     Login: guest / guest

Quick Start

Option 1: Run Everything Together (Simplest)

Start all services (RabbitMQ + Producer + Consumer):

docker compose -f docker-compose.full.yml up -d

Check the logs to see messages being sent and received:

docker compose -f docker-compose.full.yml logs -f

Stop everything:

docker compose -f docker-compose.full.yml down

Option 2: Run Services Separately (More Control)

Start just RabbitMQ:

docker compose up -d

Add the producer:

docker compose --profile producer up -d

Add the consumer:

docker compose --profile consumer up -d

Check logs:

docker compose logs -f

Stop everything:

docker compose down

Viewing and Sending Messages

Watch Consumer Process Messages

Terminal 1: Monitor consumer logs in real-time

docker compose -f docker-compose.full.yml logs -f consumer

You'll see:

[*] Waiting for messages. To exit press CTRL+C
[x] Received: Your message here
[x] Done

Send Messages Manually

Terminal 2: Send messages to the queue

# Send a single message
docker compose -f docker-compose.full.yml run --rm producer python producer.py "Hello World"

# Send message with dots (simulates longer processing - 1 second per dot)
docker compose -f docker-compose.full.yml run --rm producer python producer.py "Task with work..."

# Send multiple messages
docker compose -f docker-compose.full.yml run --rm producer python producer.py "Message 1"
docker compose -f docker-compose.full.yml run --rm producer python producer.py "Message 2"
docker compose -f docker-compose.full.yml run --rm producer python producer.py "Message 3"

Interactive Consumer (Alternative)

Run the consumer in the foreground to see output immediately:

# Stop background consumer
docker compose -f docker-compose.full.yml stop consumer

# Run consumer in foreground
docker compose -f docker-compose.full.yml run --rm consumer python consumer.py

Then send messages from another terminal and watch them process in real-time.

Management UI

Access the RabbitMQ Management Interface:

• URL: http://localhost:15672
• Username: guest
• Password: guest

View queues, messages, and connections in real-time.

Project Structure

rabbitmq-tutorial/
├── app/
│   ├── producer.py        # Sends messages to queue
│   ├── consumer.py        # Receives messages from queue
│   └── test.py           # Test without RabbitMQ
├── config/
│   └── requirements.txt   # Python dependencies (pika)
├── docker/
│   ├── Dockerfile.producer # Producer container image
│   ├── Dockerfile.consumer # Consumer container image
│   └── .dockerignore
├── docker-compose.yml     # Setup with profiles (selective start)
├── docker-compose.full.yml # Setup without profiles (start all)
└── README.md

Docker Commands Reference

Using docker-compose.full.yml (All Services)

# Start everything
docker compose -f docker-compose.full.yml up -d

# View logs (all services)
docker compose -f docker-compose.full.yml logs -f

# View logs (specific service)
docker compose -f docker-compose.full.yml logs -f producer
docker compose -f docker-compose.full.yml logs -f consumer

# Check status
docker compose -f docker-compose.full.yml ps

# Stop everything
docker compose -f docker-compose.full.yml down

Using docker-compose.yml (With Profiles)

# Start only RabbitMQ
docker compose up -d

# Start RabbitMQ + Producer
docker compose --profile producer up -d

# Start RabbitMQ + Consumer
docker compose --profile consumer up -d

# Start all services
docker compose --profile producer --profile consumer up -d

# View logs
docker compose logs -f

# Stop everything
docker compose down

Prerequisites

• Docker Desktop installed and running
• Python 3.8+ (only if running locally, not in Docker)

Troubleshooting

Docker not running:

# Start Docker Desktop first, then verify:
docker --version

See what's running:

docker compose ps

View RabbitMQ container logs:

docker logs rabbitmq

Check queue status:

• Visit http://localhost:15672
• Go to "Queues" tab
• Look for task_queue

How It Works

Producer (app/producer.py)

• Connects to RabbitMQ
• Creates a durable queue called task_queue
• Sends persistent messages
• Default message: "Hello RabbitMQ!"
• Exits after sending

Consumer (app/consumer.py)

• Connects to RabbitMQ
• Listens to task_queue
• Processes messages (simulates work by counting dots)
• Sends acknowledgment after processing
• Runs continuously waiting for messages

Message Flow

1. Producer → RabbitMQ queue
2. RabbitMQ → Consumer
3. Consumer processes → Acknowledges
4. Message removed from queue

Advanced Usage

Running Locally (Without Docker)

# Install dependencies
pip install -r config/requirements.txt

# Terminal 1 - Start consumer
python3 app/consumer.py

# Terminal 2 - Send messages
python3 app/producer.py "First message"
python3 app/producer.py "Second message..."
python3 app/producer.py "Third message....."

Messages with more dots take longer to process (1 second per dot).

Testing Without RabbitMQ

python3 app/test.py

This simulates the message flow without requiring RabbitMQ.

What You'll Learn

• Message queue concepts (producer/consumer pattern)
• Docker containerization and networking
• RabbitMQ basics (queues, durability, acknowledgments)
• Python messaging with pika library

License

MIT License - feel free to use for learning and projects.