Message bus systems are an easy way to distribute tasks.
RabbitMQ is a message broker and there are command line tools too to interact with queues.
To install these tools call:
apt-get install amqp-tools
The two main methods are:
First one is a blocking call that waits for incoming messages.
Second one is a call to put a message on a queue.
So we do have "workers" that consume a queue and "masters" that are publishing tasks.
A worker would look like:
amqp-consume -s 127.0.0.1:5672 -q "test" -e "amq.topic" --vhost "/" -r "worker1" --username=guest --password=guest -d ~/onmessage.sh
The parameters are:
Usage: amqp-consume [-dA?] [-s|--server=hostname:port] [--vhost=vhost] [--username=username] [--password=password] [-q|--queue=queue] [-e|--exchange=exchange] [-r|--routing-key=routing key] [-d|--declare] [-A|--no-ack] [-?|--help] [--usage] [OPTIONS]...
So if a message is dropped in the queue "test" and has the routing key "worker1" the command "~/onmessage.sh" is called.
The onmessage script might look like this:
nano ~/onmessage.sh && chmod +x ~/onmessage.sh
read line echo "Message: $line"
amqp-consume pipes the content of the message therefore we cannot work with parameters ($1,$2,...) but with read to save the stream into a variable called "line".
To publish a message to the "worker1" following command has to be called:
amqp-publish -e "amq.topic" -r "worker1" -b "this is a test message"
The parameters are:
Usage: amqp-publish [OPTIONS]... -e, --exchange=exchange the exchange to publish to -r, --routing-key=routing key the routing key to publish with -p, --persistent use the persistent delivery mode -C, --content-type=content type the content-type for the message -E, --content-encoding=content encoding the content-encoding for the message -b, --body=body specify the message body Connection options -s, --server=hostname:port the AMQP server to connect to --vhost=vhost the vhost to use when connecting --username=username the username to login with --password=password the password to login with
I use this pattern for a download service.
I send urls to workers that wget the target.
You can even put more than one worker on a queue - so the work load is distributed.
Or define a second queue were the workers publish results to implement asynchronous method calls.
I like the amqp-tools because they can be used with any tool or language.