Batch API + Cron = Queues

One of my favorite Drupal 6 features was the Batch API. The problem with the web is that processes are expected to be short lived. In other words, when you go to a URL, the web server is expected to process some data and return a result back to you within a short amount of time. If the web server takes too long, it usually hits a timeout limit and stops. All of this made processing large data sets, such as importing thousands of users, very problematic. Batch API solved the problem by breaking down data sets into manageable chunks that could be processed within timeout constraints.

Batch API + Cron = Queues

Once you have the ability to process large data sets like user imports, it's easy to start expanding your imagination to using batches to solve other tasks, such as sending thousands of emails for a daily newsletter. While you might assume it would be simple to use Drupal's hook_cron to run batches on a schedule, actually implementing it is difficult due to the way that Batch API makes calls to run batches. Thanks to Drupal 7's Queue API, this will not be a blog post about those difficulties. In fact, not only does Queue API solve the problem, but queues are even easier to implement than batches!

The general structure to use the Queue API is to

1. Define a Queue
2. Load the Queue
3. Process a Queue Item

Simple right? With Batch API you have to include a lot of extra code to control the batch size and looping, but with Queue API, all of that is handled for you.

Define a Queue

To define a queue, we'll need to run hook_cron_queue_info():

    /**
     * Implements hook_cron_queue_info()
     */
    function my_module_cron_queue_info() {
      $queues['my_queue'] = array(
        'worker callback' => 'my_queue_process',
        'time' => 30,
      );
      return $queues;
    }

The array key "my_queue" is simply the id for our queue, the worker callback is the function that will run on every queue item and the time is the amount of time we're allowing Drupal to call the worker.

Load the Queue

Loading items into the queue is as easy as instantiating your queue object and calling the createItem() method. Because we want to use Cron to regularly fill and process our queue, we'll do this in hook_cron()

    /**
     * Implements hook_cron()
     */
    function my_module_cron() {
      $my_queue = DrupalQueue::get('my_queue');
      foreach ($items as $item) {
        $my_queue->createItem($item);
      }
    }

To get our queue, we simply pass the queue id that was declared in hook_cron_queue_info(). The $item variable is the information we want to store in the queue.

Process a Queue Item

Because Drupal manages the looping, the worker callback that we defined earlier simply needs to deal with individual queue items.

    /**
     * Worker callback defined in hook_cron_queue_info()
     */
    function my_queue_process($item) {
      ... do something here with $item ...
    }

Since $item is exactly the same as whatever we stored to the queue, we can pass a lot of information. $item could be a user object with the user's email address and subscription information or it could be a node object that needs to be unpublished or just about anything you can think of (well anything, except for unicorns).

Cue Your Next Project With Queues

Scheduling queues with Cron is a common use case, but it's by no means the only way to use queues. Queues are amazingly flexible. What if you generated a number of membership ids, loaded them into a queue and dispensed them whenever a new user registered? Or how about loading coupon codes into a queue and providing discounts to the first 25 online customers in your store? Tell us about how you plan to use Queues in your current or upcoming project!