Job Queue
The Zend Server Job Queue component provides offline asynchronous processing of tasks and activities that can be run independently of the end user experience.
The jobs approach is used for streamlining offline processing of PHP scripts. It provides the ability to delay execution of “heavy” parts of web applications that originate from direct user interaction with a web server. As result it may significantly improve response time and reduce web server load.
Any PHP script can create a job which will be executed later, separately from a current HTTP request. The job itself is just an additional HTTP request to another PHP script that can be done on the same or different web server.
As opposed to “fire and forget” systems (like cron), Job Queue is a job management system that provides advanced capabilities, such as:
- Keeping track of batch jobs, including their status, execution time and output
- Different schedule strategies based on time, job priorities and dependencies
- Run-time statistics
- Web-based management UI
Concepts
- Job - A task defined by a URL, a set of parameters and additional properties which is to be executed by the Job Queue system. Each Job is executed once and scheduling rules can be defined to trigger periodical job creation for the same task.
- Scheduling Rule - An entity defining the periodic re-occurrence of jobs of a specific type. Each scheduling rule defines both the schedule and the job properties such as URL, parameters and priority. All jobs created by a specific scheduling rule share the same properties. this is the basis on which Recurring Jobs are created. Scheduling Rules are created either through the GUI and can also be hard-coded using an API.
- Recurring Jobs - Jobs that were generated based on a scheduling rule.
- Queues - A group of jobs grouped together to ensure easier and more efficient management and prioritization. How do queue priorities work? Zend Server will execute jobs according to the priority of the queue they are associated with. Within a queue, Zend Server will execute jobs according to the priority associated with a job.
However, there are two exceptions to this rule:
- Should Zend Server identify that the amount of jobs being executed exceeds the amount set in the queue's Max Concurrent Jobs setting (see below), the next queue in line is executed.
- Should Zend Server identify a job pending execution that has exceeded the time set in the queue's Max Job Wait Time setting (see below), this job is scheduled for immediate execution.
How does it work?
Job Queue for a single server and for a cluster are based on different architectural layouts.
Zend Server Job Queue
The Zend Server Job Queue is comprised of three main parts, the Job Queue Daemon, the Job Queue Client and one or more servers that execute the jobs.
The Job Queue Daemon
The heart of Zend Job Queue is the Job Queue Daemon – this external service takes care of managing scheduling, queuing and executing tasks. It also communicates with the Zend Server User Interface to report any errors and failures.
Jobs are successfully queued once they are written to persistent storage. This makes sure that even if the daemon has to be restarted, or even in case of complete system failure, jobs are not lost.
The Job Queue Client Extension
Jobs, be they scheduled, recurring or one-time queued jobs are all passed to the Daemon by the Job Queue Client extension. This simple PHP extension provides an easy to use API for creating and managing jobs from within PHP. It communicates with the Job Queue Daemon through a TCP socket or a UNIX domain socket.
The Job Queue Client is also a part of the Zend Server Administration Interface – which, through the same extension, allows the end user to create and manage jobs from the convenience of a web GUI.
Executing Web Servers
As far as execution is concerned, running a job is no more than sending a simple HTTP request. When creating a new job through API or GUI, the user specifies the URL for this HTTP request, along with some possible optional parameters. When it's time to run the job, the Job Queue Daemon sends an HTTP request to the server specified by the URL along with the provided headers and parameters, and waits for a response.
The executing HTTP server can be any HTTP server – it might even be the same server that queued the job. This design makes it very easy to manage the system. You don't have to (but can, if needed) set up dedicated servers for handling off-line jobs. You don't have to (but can, if needed) create and manage a special PHP configuration for this environment. You can reuse existing code to a level where sometimes the difference between off-line and on-line execution is decided based on a simple request parameter.
Zend Server Cluster Job Queue
The Zend Server Cluster Job Queue is comprised of four main components:
-
Web Servers- The servers in which the PHP and Job Queue Extensions reside.
In some cases, these servers execute jobs as well. In other case, jobs are executed on Backend Web servers. -
Job Queue Extensions- A PHP extension that passes the jobs to the Job Queue Daemons.
-
Job Queue Daemons- The component that is in charge of managing scheduling, queing and executing tasks.
-
MySQL Data Base- The component that stores the jobs created throughout the cluster.
Web Servers
The Web servers hold the PHP and the Job Queue Extensions, and receive HTTP requests from the Job Queue Daemons.
The Job Queue Extension
Jobs, be they scheduled, recurring or one-time queued jobs are all passed to the Daemon by the Job Queue Client extension. This simple PHP extension provides an easy to use API for creating and managing jobs from within PHP. It communicates with the Job Queue Daemon through a TCP socket or a UNIX domain socket.
The Job Queue Client is also a part of the Zend Server Administration Interface – which, through the same extension, allows the end user to create and manage jobs from the convenience of a web GUI.
The Job Queue Daemons
The heart of Zend Job Queue is the Job Queue Daemon – this external service takes care of managing scheduling, queuing and executing tasks. It also communicates with the Zend Server User Interface to report any errors and failures. The Job Queue Daemons receive the job details from the Job Queue Extensions over a local connection (a TCP or UNIX domain socket). They then store them in the MySQL DB, while checking for additional jobs to be executed in the DB. Jobs are successfully queued once they are written to persistent storage. This makes sure that even if the daemon has to be restarted, or even in case of complete system failure, jobs are not lost.
The MySQL DB Server
A Zend Server cluster uses a MySQL DB server that serves its various components and holds the Job Queue tables. When a node is attached to the cluster, its Job Queue Daemon is directed to use this server and begin submitting and executing jobs through it. The default setup uses a MySQL DB server utilizing InnoDB, the leading transactional storage engine for the MySQL open source database. Because this storage engine is ACID compliant and transactional the default setup of MySQL in the clustered configuration is already highly available and fault tolerant. Other MySQL configurations are possible, please consult MySQL experts for other configuration that are compatible with Zend Server’s usage patterns.
Further Information
In clustered setups, the Job Queue component uses a MySQL DB as a storage backend. Consequently, performance depends on MySQL's configuration settings.
For a deployment that requires high-performance job queuing, you should consider modifying MySQL’s strategy for flushing data to disk. This can be done by changing the 'innodb_flush_log_at_trx_commit' directive value from 1 to 2, when:
1= Flush data to disk every time a transaction is committed (default).
2= Flush data to disk every second.
For more information, please refer to Oracle / MySQL documentation.