How To Run Resque Workers On AWS Elastic Beanstalk

A Quick Guide to Running Resque

Diana

Diana

Ruby Developer at Softvision
Ruby Developer at Softvision with 5 years experience in web programming who also enjoys playing ops. Linux registered user: #586145 https://www.linuxcounter.net. I am also a dog lover who enjoys playing piano and reading fiction.
Diana

Latest posts by Diana

This article aims to provide a detailed description of how to setup Resque and run workers on an AWS worker tier environment, considering all the available hooks which Elastic Beanstalk offers.

First, let’s start with a brief introduction to AWS and Resque:

Amazon Web Services is one of the biggest cloud computing platforms, providing different types of services such as infrastructure as a service, platform as a service and software as a service. One of these services is Elastic Beanstalk, which we’ll cover in this article.

Elastic Beanstalk allows you to easily deploy and manage applications in AWS without worrying about the infrastructure that runs those applications. It handles load balancing, auto-scaling, and health monitoring. For more details, you can refer to the AWS Elastic Beanstalk documentation.

Resque is a Ruby library for creating and handling background jobs, built on top of Redis. It enables you to create jobs and place them in a queue, which, afterward, will be processed. Other alternatives for running background jobs are DelayedJob, Sidekiq or SuckerPunch.

AWS Elastic Beanstalk supports a number of different worker types and we decided to use Resque since it was integrated into our project before the AWS migration.

1. Create a worker tier environment

Navigate to the Elastic Beanstalk page and select Create Environment from the Actions drop-down. From the modal that pops up, choose Worker environment.

You will be taken to a setup page, where you must provide some environment information like name, subdomain and description. In the Base configuration section, select the Platform to be Ruby and click Configure more Options from the bottom of the page.

In the Configuration page, the first thing you’ll need to do is to select the third option (Custom configuration) from Configuration presets section. This enables you to change your Ruby version along with the web server you want. Usually, we do not need a web server running on a worker environment since its purpose is only to run Resque for processing workers and not to handle HTTP requests. However, for running periodic tasks (cron jobs), it’s necessary to have a web server because the worker daemon performs a POST request to the instance in order to trigger a specific scheduled job. But this is a topic for another discussion. Until then, you can refer to the Amazon documentation on how to define periodic tasks.

You can now move on and configure your environment further as needed. You can also refer to this link for a detailed documentation on creating a new environment in Elastic Beanstalk.

2. Setup Resque

Now we need to have a Resque process that will run all our queues (or maybe multiple processes, each handling different queues).

Usually, to start a worker we would simply need to run the following command:

bundle exec rake resque:work QUEUES=*

If we need multiple workers we would have something like this:

bundle exec rake resque:work QUEUES=high,medium
bundle exec rake resque:work QUEUES=low,emails

Now, all we need to do is run the above commands when we deploy our application and we should be good to go. But usually, when things sound too good to be true, something weird is going on. I’ll explain.

Start Resque as a daemon

What if we were to need multiple Resque processes, each one running different workers in the background and we would also like for each worker to write to its own log file, so we can debug if any issues occur? This means that multiple commands need to be executed. So it would be a lot easier if we had a Resque service which we can start as a Linux daemon like this:

  • service resque start/stop/restart

In order to be able to run Resque this way, we must manage it through a shell script. Let’s simply call it resque_service.sh and save it somewhere in our project. When the code is deployed, we need to make sure the script is copied to /etc/init.d. Here’s how the script looks like:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          Resque
# Description:       Starts 4 resque processes, each of them handling different workers and writes an output in /var/log/workers directory
### END INIT INFO
PATH=/opt/rubies/ruby-current/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
 RUNAS=webapp

APPDIR=/var/app/current/
LOGDIR=/var/log/workers/
PIDDIR=/var/run/resque/

mkdir -p "$LOGDIR"
chown -R "$RUNAS":"$RUNAS" "$LOGDIR"
mkdir -p "$PIDDIR"
chown -R "$RUNAS":"$RUNAS" "$PIDDIR"
PIDFILE_1="$PIDDIR"resque_1.pid
PIDFILE_2="$PIDDIR"resque_2.pid
PIDFILE_3="$PIDDIR"resque_3.pid
PIDFILE_4="$PIDDIR"resque_4.pid
FIRST_WORKER="PIDFILE=$PIDFILE_1 BACKGROUND=yes TERM_CHILD=1 LOGGING=1 /opt/rubies/ruby-current/bin/bundle exec /opt/rubies/ruby-current/bin/rake resque:work QUEUES=first_worker_queue"
SECOND_WORKER="PIDFILE=$PIDFILE_2 BACKGROUND=yes TERM_CHILD=1 LOGGING=1 /opt/rubies/ruby-current/bin/bundle exec /opt/rubies/ruby-current/bin/rake resque:work QUEUES=second_worker_queue"
THIRD_WORKER="PIDFILE=$PIDFILE_3 BACKGROUND=yes TERM_CHILD=1 LOGGING=1 /opt/rubies/ruby-current/bin/bundle exec /opt/rubies/ruby-current/bin/rake resque:work QUEUES=third_worker_queue"
FOURTH_WORKER="PIDFILE=$PIDFILE_4 BACKGROUND=yes TERM_CHILD=1 LOGGING=1 /opt/rubies/ruby-current/bin/bundle exec /opt/rubies/ruby-current/bin/rake resque:work QUEUES=fourth_worker_queue"
LOGFILE_1="$LOGDIR"first_worker.log
LOGFILE_2="$LOGDIR"second_worker.log
LOGFILE_3="$LOGDIR"third_worker.log
LOGFILE_4="$LOGDIR"fourth_worker.log
 start() {
    . /opt/elasticbeanstalk/support/envvars
if [ "$MAINTENANCE_MODE" ]; then
     echo "Resque not starting..." >&2
     echo "Maintenance mode is enabled" >&2
     return 0

   fi

   if [ -f "$PIDFILE_1" ] && kill -0 $(cat "$PIDFILE_1"); then
     echo 'Service already running' >&2
     return 1
   fi
   echo 'Starting service…' >&2
   cd "$APPDIR"
   local CMD="$FIRST_WORKER &> \"$LOGFILE_1\""
   su -s /bin/bash -c "$CMD" $RUNAS
   local CMD="$SECOND_WORKER &> \"$LOGFILE_2\""
   su -s /bin/bash -c "$CMD" $RUNAS
   local CMD="$THIRD_WORKER &> \"$LOGFILE_3""
   su -s /bin/bash -c "$CMD" $RUNAS
   local CMD="$FOURTH_WORKER &> \"$LOGFILE_4\""
   su -s /bin/bash -c "$CMD" $RUNAS
   echo 'Service started' >&2
}

 stop() {
   if [ ! -f "$PIDFILE_1" ] || ! kill -0 $(cat "$PIDFILE_1"); then
     echo 'Service not running' >&2
     return 1
   fi
   echo 'Stopping service…' >&2
   kill -15 $(cat "$PIDFILE_1") && rm -f "$PIDFILE_1"
   kill -15 $(cat "$PIDFILE_2") && rm -f "$PIDFILE_2"
   kill -15 $(cat "$PIDFILE_3") && rm -f "$PIDFILE_3"
   kill -15 $(cat "$PIDFILE_4") && rm -f "$PIDFILE_4"
   echo 'Service stopped' >&2
}

case "$1" in

   start)
     start
     ;;
   stop)
     stop
     ;;
   restart)
     stop
     start
     ;;
   *)
     echo "Usage: $0 {start|stop|restart}"
esac

It is very important to set the PATH environment variable at the beginning of the script because running the script with sudo resets it.
After the above file is copied to /etc/init.d/ we can make use of it like this:

  • sudo service resque start/stop/restart

Note:
MAINTENANCE_MODE environment variable comes in handy when you don’t want Resque to start. Simply set it to true and you’re all set.

Elastic Beanstalk hooks

What we ultimately want to achieve is:

  • Start resque service when we deploy our code
  • Restart resque service when we restart the application server
  • Restart resque service when we change software configuration variables

To be able to do this, we will make use of Elastic Beanstalk hooks. These hooks are actually scripts that are run in response to management operations (see a detailed explanation on AWS documentation). For example, when we deploy our code, all hooks from the directory /opt/elasticbeanstalk/hooks/appdeploy/” get executed. This means that Resque must also be executed as a hook, available within this directory.

As previously explained, we need to implement 3 hooks, so we will create a configuration file in .ebextensions folder within our project. Let’s name it resque.config:

commands:
  create_post_dir:
      command: "mkdir -p /opt/elasticbeanstalk/hooks/appdeploy/post/"
      ignoreErrors: true
  create_restart_dir:
      command: "mkdir -p /opt/elasticbeanstalk/hooks/restartappserver/post/"
      ignoreErrors: true
  create_restart_dir:
      command: "mkdir -p /opt/elasticbeanstalk/hooks/configdeploy/post/"
      ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/99_resque_starter.sh":
   mode: "000755"
   owner: root
   group: root
   content: |
     cp /var/app/current/continuous_delivery/resque_service.sh /etc/init.d/resque
     chmod +x /etc/init.d/resque
     service resque restart
"/opt/elasticbeanstalk/hooks/restartappserver/post/02_restart_resque.sh":
   mode: "000755"
   owner: root
   group: root
   content: |
     cp /var/app/current/continuous_delivery/resque_service.sh /etc/init.d/resque
     chmod +x /etc/init.d/resque
     service resque restart
"/opt/elasticbeanstalk/hooks/configdeploy/post/02_restart_resque.sh":
   mode: "000755"
   owner: root
   group: root
   content: |
     cp /var/app/current/continuous_delivery/resque_service.sh /etc/init.d/resque
     chmod +x /etc/init.d/resque
     service resque restart

Each command will create a file needed for a specific action. Each file that will be executed as root will copy the resque_service.sh file that we previously created into /etc/init.d/ directory and will give execute permissions to start/restart the service.

That’s it. When the code is deployed, Elastic Beanstalk searches for the .ebextensions folder for configurations and scripts.

Note
If you are deploying a web environment together with the worker one, you definitely don’t want to have those hooks executed in the web environment. So it is better to deploy the web app first, and then copy the resque.config file (from another location in your project) into the .ebextensions folder. This way, only the worker environment will have access to the Resque service and hooks.

Conclusion

While there are other alternatives for processing background jobs, Resque is a library that can easily be integrated in your application and it can also be used with the resque-web ruby gem which is a user-friendly interface to the Resque job queue system.

Hope this article was useful. If you have any questions or suggestions don’t hesitate to send me an email at diana.manolea@softvision.ro

Share This Article


Diana

Diana

Ruby Developer at Softvision
Ruby Developer at Softvision with 5 years experience in web programming who also enjoys playing ops. Linux registered user: #586145 https://www.linuxcounter.net. I am also a dog lover who enjoys playing piano and reading fiction.
Diana

Latest posts by Diana

No Comments

Post A Comment