Skip to content

Notification architecture

Jump to bottom Edit New page
Simon Walker edited this page Mar 14, 2022 · 2 revisions

Notifications are delivered via the AMQP 0-9-1 protocol to a compatible message broker. Quite what happens to notifications after that point is not the concern of the ACC application itself, so this isn't going to be covering a development perspective of how they work. This document is mostly from an operational perspective.

Overview

We're using RabbitMQ as the message broker, and to account for network issues/server restarts/etc, we're using a pair of brokers. One broker hosted on WMCS receives the messages from the local instances of the application. The other broker is hosted on the same infrastructure as Helpmebot. We use a shovel to move messages between brokers.

To quickly summarise how AMQP message brokers work, I'll introduce some basic concepts. We connect to the broker (via TCP), and then "publish" our message to an "exchange". We can also read messages from a "queue". Over-simplifying, exchanges are write-only, and queues are read-only. A message is published with a "routing key" - this can be any string value. Queues and exchanges are connected together with "bindings", which can be configured to send specific messages to different queues based on the routing key of the message.

For ACC, the application writes to a single exchange (acc.notification) on the WMCS broker with a configured routing key. This exchange has a single binding to a queue (acc.notification.queue) which receives all messages published to the exchange (this exchange is a fanout exchange).

Once messages are on the queue, they are picked up by a shovel - basically a thing which reads messages off one queue and writes them to an exchange somewhere else. This shovel is configured to read messages from acc.notification.queue and write them to an exchange on Helpmebot's message broker (helpmebot.prod.x.notification).

On Helpmebot's side, the exchange helpmebot.prod.x.notification is configured as a direct exchange, so it uses the routing key on the message to match the routing key filters on the bindings to route to the "correct" queue. In practice, there's only a single queue, and it's used as a filter to only route messages to the queue which are for a known set of routing keys.

Helpmebot then reads the messages from the queue and delivers the content directly to IRC, using the message routing key as a channel name.

Quick-start for developers

Note - the configuration described below is not secure.

  • Install RabbitMQ on the same server ACC runs on (Debian: sudo apt-get install rabbitmq-server; Docker: use image rabbitmq:3-management mapping ports 5672 and 15672).
  • Enable the management UI (sudo rabbitmq-plugins enable rabbitmq_management, docker users can skip)
  • Log into the management UI on port 15672 (guest/guest)
  • Create a queue with any name you want, eg potato from the RabbitMQ management UI
  • Ensure ACC's configuration setting $amqpConfiguration is set to the default (unless you've started changing more things in the RabbitMQ management UI; at that point you're on your own)
  • Edit the domain setting for notification target to the queue name on RabbitMQ (eg, potato). Also set $ircBotNotificationRoutingKey = 'potato; in configuration.
  • Ensure $ircBotNotificationsEnabled = 1;
  • You should be able to use the RabbitMQ management UI to view and acknowledge (thus dequeue) items on the queue as they're generated.

The technical stuff: this config should work because the default exchange "" acts as a direct exchange with implicit bindings on every queue, so the routing key specifies the target queue without further configuration. We don't do this in prod, but it should work fine for a local dev instance.

Configuration

  • Install RabbitMQ on WMCS. Enable the management and shovel plugins: sudo rabbitmq-plugins enable rabbitmq_management rabbitmq_shovel_management

  • Add a new user for yourself:

sudo rabbitmqctl add_user myusername myp@ssw0rd
sudo rabbitmqctl set_permissions -p / myusername '.*' '.*' '.*'
sudo rabbitmqctl set_user_tags myusername administrator
  • Set up an SSH tunnel mapping port 15672 to your local machine.
  • Browse to http://localhost:15672 (or equivalent), and log in.
  • Make sure the guest user is deleted.

WMCS configuration summary:

  • Create exchange acc.notification, type fanout, durable, autodelete=no.

  • Create exchange acc.deadletter, type fanout, durable, autodelete=no.

  • Create queue acc.notification.queue, durable

  • Create queue acc.deadletter.queue, durable, lazy mode

  • Bind acc.deadletter to acc.deadletter.queue with blank routing key

  • Bind acc.notification to acc.notification.queue with blank routing key

  • Create policy to set the dead letter target and set some time/length limits on the queue.

    • pattern: ^acc\.notification\.queue$
    • apply to queues
    • dead-letter-exchange: acc.deadletter
    • max-length: 20
    • message-ttl: 900000

  • Create user ExampleUsername. THIS MUST MATCH the username on Helpmebot's broker, though the password need not match.

    • Grant access to vhost / with configure ^$, read ^$, write ^acc\.notification$. This permits this user only to write to one exchange.

  • Create another user for the shovel - I used shovel.

    • Grant access to vhost / with configure ^$, read ^acc\.notification\.queue$, write ^$. This permits this user only to read from one queue.

  • Configure ACC to connect to the broker with username ExampleUsername, writing to exchange acc.notification on vhost /

  • Configure a shovel:

    • Name: whatever you want
    • Source:
      • AMQP 0.9.1
      • amqp://shovel@
      • Queue: acc.notification.queue
      • Prefetch count left blank
      • Auto-delete: never
    • Destination:
      • AMQP 0.9.1
      • amqps://ExampleUsername:[redacted]@mq.srv.stwalkerster.net (ensure password is URL-encoded)
      • Exchange: helpmebot.prod.x.notification
      • Forwarding headers: Yes
      • Routing key left blank
    • Reconnect delay left blank
    • Ack mode: on publish

Helpmebot configuration summary:

  • Create a user for Helpmebot, with configure/write/read to ^helpmebot\.prod\.[xq]\..*$
  • Configure Helpmebot to point to the broker with a prefix of helpmebot.prod, and start it. It'll create most of the queues/exchanges it needs itself.
  • Create user ExampleUsername. THIS MUST MATCH the username on Helpmebot's broker, though the password need not match.
    • Grant access to vhost / with configure ^$, read ^$, write ^helpmebot\.prod\.x\.notification$. This permits this user only to write to one exchange.
  • Ensure the necessary bindings are set up between helpmebot.prod.x.notification and helpmebot.prod.q.notification. This can be managed with bot commands.

Notes:

  • Helpmebot only accepts messages with a user_id set.
  • The shovel only works for messages with user_id == ExampleUsername.
  • It's not possible to set user_id to a value different from you current username.
Add a custom footer
Add a custom sidebar
Clone this wiki locally