Monitoring Rules JSON Structure

ZendHQ monitoring rules define conditions that trigger monitoring events and optional actions that will be executed when a monitoring event is triggered.

ZendHQ monitoring rules are a JSON document with an array of zero to many monitoring rules.

If defined in /opt/zend/zendphp/etc/default_monitor_rules.json prior to initialization, that document will define the default rules for the ZendHQ instance. Otherwise, you may provide an array of rules as the configuration "value" when setting monitoring rules via the websocket API as follows:

{"id":1,"method":"conf.set","params":{"ns":"monitor","key":"rules","value":[...]}}

Monitoring rule

A monitoring rule must contain the following members:

Member Description
rule_id A positive integer value uniquely identifying the rule
rule_type A string with the rule type
rule_name A string with the human-readable rule name
conditions An array with rule conditions

A monitoring rule can contain the following optional members:

Member Description
rule_description A string with the human-readable description of the rule
actions An array of actions

Monitoring rules of the types function-slow-exec and function-error must contain the following additional members:

Member Description
regex_match A regular expression to watch specific PHP functions

rule_id

The rule_id member is a positive integer value (zero included) uniquely identifying the rule.

rule_type

The rule_type member specifies the type of the rule and must be one of the following:

Value Description
function-slow-exec A PHP function took too long time to execute
function-error A PHP function returned false
request-slow-exec A request took too long time to execute
request-high-mem-usage A request used too much memory
php-error A PHP error occurred
custom A custom event was generated in the PHP script
  • A function-slow-exec rule is triggered when a watched function takes longer to execute than a specified threshold.

  • A function-error rule is triggered when a watched function returns false; these are generally used to monitor PHP internal functions that use a return value of false to indicate an error.

  • A request-slow-exec rule is triggered when a request takes longer to execute than a specified threshold.

  • A request-high-mem-usage rule is triggered when more memory than a specified threshold is in use at the end of the request.

  • A php-error rule is triggered when a PHP error matching a given PHP error mask occurs.

  • A custom rule is triggered when the event is generated in the PHP script. See Custom Monitoring Events for details how custom events are generated in the PHP script.

rule_name

The rule_name member is a string uniquely identifying the rule.

conditions

The conditions member is an array of rule conditions. The array must contain at least one condition.

rule_description

The rule_description is a human-readable string describing the rule.

actions

The actions member is an array of actions.

regex_match

The regex_match member is a regular expression used to match the rule against specific PHP internal or user functions. The rule is applied only if the name of the PHP function or method name matches the regular expression. Specify method names in the format {Class Name}::{Method Name} (for example, PDO::connect).

See https://github.com/google/re2/wiki/Syntax for syntax.

Monitoring rule conditions

Monitoring rule conditions specify the criteria for monitoring events. Monitoring rule conditions depend on the type of the rule, but must always contain the following members:

Member Description
condition_id A positive integer value uniquely identifying the condition
severity One of "critical", "warning", or "notice"

Monitoring rule conditions for rules of the types function-slow-exec, request-slow-exec, and request-high-mem-usage must contain the following additional member:

Member Description
threshold A positive number value

Monitoring rule conditions for the php-error type must contain the following additional member:

Member Description
mask PHP error mask

condition_id

The condition_id member is a positive integer value (zero included) uniquely identifying the condition within the rule.

severity

The severity member specifies the severity of the monitoring event generated when the condition is met.

threshold

The threshold member specifies the threshold at which the monitoring event with the given severity is generated.

For monitoring rules of the types function-slow-exec and request-slow-exec, the threshold value is a time in milliseconds.

For the request-high-mem-usage monitoring rule, the threshold value is a number of bytes in use at the end of the request.

mask

The mask member specifies a bitmask with PHP error types that generate the monitoring event with the given severity.

The mask can have the following values combined with a bitwise OR operation:

Value PHP error type
0x0001 ERROR
0x0002 WARNING
0x0004 PARSE ERROR
0x0008 NOTICE
0x0010 CORE ERROR
0x0020 CORE WARNING
0x0040 COMPILE ERROR
0x0080 COMPILE WARNING
0x0100 USER ERROR
0x0200 USER WARNING
0x0400 USER NOTICE
0x0800 STRICT
0x1000 RECOVERABLE ERROR
0x2000 DEPRECATED
0x4000 USER DEPRECATED

Monitoring rule actions

Monitoring rule actions specify actions that are executed when a monitoring event is generated.

Monitoring rule actions must contain the following members:

Member Description
action_id A positive integer value uniquely identifying the action
action_type One of "url" or "email"
action_target A string specifying the target of the action

action_id

The action_id is a positive integer value (zero included) uniquely identifying the action within the monitoring rule.

action_type

The action_type member specifies the type of the action and must have one of the following values:

Value Description
url A webhook that receives notifications for monitoring events
codetrace Triggers the code trace collection and storage

The url action performs an HTTP POST request with the monitoring event data to the given URL. See ZendHQ Monitoring Event Data for monitoring event data details.

action_target

The action_target member specifies the target of the action.

For "url" actions this is the url for the HTTP POST request.

ZendHQ Monitoring Event Data

If a monitoring rule contains monitoring event actions and the event is generated, then a JSON document with monitoring event data is included in the action.

The JSON document contains the following members:

Member Description
event_id A number uniquely identifying the event
issue_id A number uniquely identifying the issue
request_id A number uniquely identifying the request
name Name of the monitoring rule
type Type of the monitoring rule
severity One of "critical", "warning", or "notice"
time_sec UNIX time of the event as seconds since the EPOCH
request JSON object with request details
function Optional JSON object with PHP function details
error Optional JSON object with PHP error details
stack_trace Optional JSON array with stack trace frames
custom Optional JSON object with custom event details
variables Optional JSON object with server variables
duration_sec Optional number of seconds
memory_usage_bytes Optional number of bytes

event_id

The event_id member contains the ID of the monitoring event.

issue_id

The issue_id member contains the ID of the monitoring issue.

If event aggregation is enabled, then multiple monitoring events can have the same issue ID value.

request_id

The request_id member contains the ID of the request that caused the monitoring event.

If multiple monitoring events are generated during a request, they all have the same request_id value.

name

The name member contains the name of the monitoring rule (see rule_name) that triggered the event.

type

The type member contains the type of the monitoring rule (see rule_type) that triggered the event.

severity

The severity member is the severity of the monitoring event.

time_sec

The time_sec member contains the UNIX time of the event as a number of seconds since the EPOCH (January 1, 1970 midnight UTC/GMT).

request

The request member contains request details and has the following members:

Member Description
url The requested target url
php_version The PHP version string, for example, "8.0.19"
node_name Name of the PHP node (uname or custom)
pid PID of the PHP process that processed the request

function

The optional function member contains PHP function details and has the following members:

Member Description
function_name Name of the PHP function
function_args Optional JSON array with function argument values
file_name Optional PHP script file name
line_no Optional line number in the PHP script file

error

The optional error member contains PHP error details and has the following members:

Member Description
message The PHP error message text
error_type The PHP error type as a numeric value
error_type_str The PHP error type as a text
file_name PHP script file name where the error occurred
line_no Line number in the PHP script file where the error occurred

stack_trace

The optional stack_trace member contains a JSON array with the PHP stack trace frames. Stack trace frames in the array contain the following members:

Member Description
function_name Name of the PHP function
file_name Optional PHP script file name
line_no Optional line number in the PHP script file

custom

The optional custom member contains custom event details and has the following members:

Member Description
type A type string
text A text
user_data User data value

See Custom Monitoring Events for details.

variables

The optional variables member contains server variables and their values, such as the content of the $_SERVER variable. See PHP values for the details how PHP values are stored in JSON documents.

duration_sec

The optional duration_sec member contains the duration of a function or a request execution that triggered the event as a number of seconds.

memory_usage_bytes

The optional memory_usage_bytes member contains the number of used memory that triggered the event as a number of bytes.

Sample monitoring event data

{
    "event_id": 95,
    "issue_id": 95,
    "memory_usage_bytes": 49166992,
    "name": "High Memory Usage",
    "request": {
        "node_name": "node1",
        "php_version": "8.1.5",
        "pid": 68381,
        "url": "http://example.com/monitor.php?high_mem_usage"
    },
    "request_id": 321,
    "severity": "warning",
    "time_sec": 1652341693.624875,
    "type": "request-high-mem-usage",
    "variables": {
        "_REQUEST": {
            "high_mem_usage": ""
        },
        "_SERVER": {
            "DOCUMENT_ROOT": "/srv/http",
            "HTTP_ACCEPT": "text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8",
            "HTTP_ACCEPT_ENCODING": "gzip, deflate",
            "HTTP_ACCEPT_LANGUAGE": "en-GB,en;q=0.9",
            "HTTP_CONNECTION": "keep-alive",
            "HTTP_HOST": "example.com",
            "HTTP_REFERER": "http://example.com/monitor.php",
            "HTTP_UPGRADE_INSECURE_REQUESTS": "1",
            "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit
                                /605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15",
            "PHP_SELF": "/monitor.php",
            "QUERY_STRING": "high_mem_usage",
            "REMOTE_ADDR": "127.0.0.1",
            "REMOTE_PORT": "51914",
            "REQUEST_METHOD": "GET",
            "REQUEST_TIME": 1652341693,
            "REQUEST_TIME_FLOAT": 1652341693.463072,
            "REQUEST_URI": "/monitor.php?high_mem_usage",
            "SCRIPT_FILENAME": "/srv/http/monitor.php",
            "SCRIPT_NAME": "/monitor.php",
            "SERVER_NAME": "127.0.0.1",
            "SERVER_PORT": "80",
            "SERVER_PROTOCOL": "HTTP/1.1",
            "SERVER_SOFTWARE": "PHP 8.1.5 Development Server"
        }
    }
}

PHP values

The following rules are applied when PHP values are written into JSON documents:

PHP value JSON value
string string
integer number
float number
boolean boolean
array array
object object with public property values from the PHP object
any other null

Sample Monitoring Rules

Slow function execution rule

This rule triggers when the exec() PHP internal function or the slow_function() PHP user function takes too long to execute.

A critical monitoring event is generated if they take 1000 or more milliseconds and a warning monitoring event if they take 500 or more milliseconds to execute.

{
    "rule_id":  1,
    "rule_type":    "function-slow-exec",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "Slow Function Execution",
    "description":  "Triggered when one of the watched functions runs longer than the specified duration",
    "regex_match": "exec|slow_function",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 1000
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 500
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Function error rule

This rule triggers when the fopen() PHP internal function or the function_that_fails() PHP user function return false.

The severity of the monitoring event is a warning.

{
    "rule_id": 2,
    "rule_type": "function-error",
    "rule_parent_id": -1,
    "app_id": -1,
    "rule_name": "Function Error",
    "description": "Triggered when one of the watched functions returns FALSE",
    "regex_match": "fopen|function_that_fails",
    "conditions": [
        {
            "condition_id": 1,
            "severity": "warning"
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Slow request execution rule

This rule triggers when the PHP request takes too long to execute.

A critical monitoring event is generated if it takes 5000 or more milliseconds and a warning monitoring event if it takes 2000 or more milliseconds to execute.

{
    "rule_id":  3,
    "rule_type":    "request-slow-exec",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "Slow Request Execution",
    "description":  "Triggered when a PHP request's runtime is longer than the specified duration",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 5000
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 2000
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

High memory usage rule

This rule triggers when a PHP request uses too much memory.

A critical monitoring event is generated when 49152 or more bytes of memory is in use and a warning monitoring event is generated when 24576 or more bytes of memory is in use at the end of the request.

{
    "rule_id":  4,
    "rule_type":    "request-high-mem-usage",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "High Memory Usage",
    "description":  "Triggered when a PHP request consumes more than the specified amount of memory",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 49152
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 24576
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

PHP error rule

This rule triggers when one of the PHP errors specified in the conditions occurs.

A critical monitoring event is generated if the PHP error type matches the bitmask 0x0055 (E_ERROR, E_PARSE_ERROR, E_CORE_ERROR, and E_COMPILE_ERROR) and a warning monitoring event is generated if the PHP error type matches the bitmask 0x17A2 (E_WARNING, E_CORE_WARNING, E_COMPILE_WARNING, E_USER_ERROR, E_USER_WARNING, E_USER_NOTICE, E_RECOVERABLE_ERROR).

{
    "rule_id":  5,
    "rule_type":    "php-error",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "PHP Error",
    "description":  "Triggered when a PHP error of one of the selected error levels is reported",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "mask": 85
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "mask": 6050
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Custom event rule

This rule triggers when the monitor_custom_event() function or the monitor_custom_event_ex($ruleName = "Custom Event") function are called in the PHP script.

The severity of the monitoring event is a warning unless a specific value is given as the $userSeverity argument when calling the monitor_custom_event() function.

{
    "rule_id": 6,
    "rule_type": "custom",
    "rule_parent_id": -1,
    "rule_name" : "Custom Event",
    "description": "Triggered when the API function 'monitor_custom_event' is called",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "warning"
        }
    ]
}