Zend Job Queue

Job Queues provide offline asynchronous processing of tasks and activities that can be run independently of the end user experience.

The Jobs approach is used for streamlining offline processing of PHP scripts. It provides the ability to delay execution of “heavy” parts of web applications that originate from direct user interaction with a Web Server. As result it may significantly improve response time and reduce Web Server load.

Any PHP script can create a job which will be executed later, separately from a current HTTP request. The job itself is just an additional HTTP request to another PHP script that can be done on the same or different Web Server.

As opposed to “fire and forget” systems,(like cron), the Job Queue is a job management system that provides advanced capabilities such as:

Keeping track of batch jobs, including their status, execution time and output
Different schedule strategies based on time, job priorities and dependencies
Run-time statistics
Web-based management GUI Graphical user interface

The functional diagram of the Job Queue is shown on the following picture.

Concepts and Terminology

Job - a task defined by a URL, a set of parameters and additional properties which is to be executed by the Job Queue system. Each Job is executed once and scheduling rules can be defined to trigger periodical job creation for the same task.
Scheduling Rule - an entity defining the periodic re-occurrence of jobs of a specific type. Each scheduling rule defines both the schedule and the job properties such as URL, parameters and priority. All jobs created by a specific scheduling rule share the same properties. this is the basis on which Recurring Jobs are created. Scheduling Rules are created either through the GUI and can also be hard-coded using an API.
Recurring Jobs - are jobs that were generated based on a scheduling rule

Architectural Overview

The Zend job queue is comprised of three main parts, the Job Queue Daemon, the Job Queue Client and one or more servers that execute the jobs.

The Job Queue Daemon

The heart of Zend Job Queue is the Job Queue Daemon – this external service takes care of managing scheduling, queuing and executing tasks. It also communicates with the Zend Server Administration Interface to report any errors and failures.

Jobs are successfully queued once they are written to persistent storage. This makes sure that even if the daemon has to be restarted, or even in case of complete system failure, jobs are not lost.

The Job Queue Client Extension

Jobs, be they scheduled, recurring or one-time queued jobs are all passed to the Daemon by the Job Queue Client extension. This simple PHP extension provides an easy to use API for creating and managing jobs from within PHP. It communicates with the Job Queue Daemon through a TCP socket or a UNIX domain socket.

The Job Queue Client is also a part of the Zend Server Administration Interface – which, through the same extension, allows the end user to create and manage jobs from the convenience of a web GUI.

Executing Web Servers

As far as execution is concerned, running a job is no more than sending a simple HTTP request. When creating a new job through API or GUI, the user specifies the URL for this HTTP request, along with some possible optional parameters. When it's time to run the job, the Job Queue Daemon sends an HTTP request to the server specified by the URL along with the provided headers and parameters, and waits for a response.

The executing HTTP server can be any HTTP server – it might even be the same server that queued the job. This design makes it very easy to manage the system. You don't have to (but can, if needed) set up dedicated servers for handling off-line jobs. You don't have to (but can, if needed) create and manage a special PHP configuration for this environment. You can reuse existing code to a level where sometimes the difference between off-line and on-line execution is decided based on a simple request parameter.