Monitoring Rules JSON Structure

ZendHQ monitoring rules define conditions that trigger monitoring events and optional actions that will be executed when a monitoring event is triggered.

ZendHQ monitoring rules are a JSON document with an array of zero to many monitoring rules.

If defined in /opt/zend/zendphp/etc/default_monitor_rules.json prior to initialization, that document will define the default rules for the ZendHQ instance. Otherwise, you may provide an array of rules as the configuration "value" when setting monitoring rules via the websocket API as follows:

{"id":1,"method":"conf.set","params":{"ns":"monitor","key":"rules","value":[...]}}

Monitoring rule

A monitoring rule must contain the following members:

A monitoring rule can contain the following optional members:

Monitoring rules of the types function-slow-exec and function-error must contain the following additional members:

rule_id

The rule_id member is a positive integer value (zero included) uniquely identifying the rule.

rule_type

The rule_type member specifies the type of the rule and must be one of the following:

• A function-slow-exec rule is triggered when a watched function takes longer to execute than a specified threshold.

• A function-error rule is triggered when a watched function returns false; these are generally used to monitor PHP internal functions that use a return value of false to indicate an error.

• A request-slow-exec rule is triggered when a request takes longer to execute than a specified threshold.

• A request-high-mem-usage rule is triggered when more memory than a specified threshold is in use at the end of the request.

• A php-error rule is triggered when a PHP error matching a given PHP error mask occurs.

• A custom rule is triggered when the event is generated in the PHP script. See Custom Monitoring Events for details how custom events are generated in the PHP script.

rule_name

The rule_name member is a string uniquely identifying the rule.

conditions

The conditions member is an array of rule conditions. The array must contain at least one condition.

rule_description

The rule_description is a human-readable string describing the rule.

actions

The actions member is an array of actions.

regex_match

The regex_match member is a regular expression used to match the rule against specific PHP internal or user functions. The rule is applied only if the name of the PHP function or method name matches the regular expression. Specify method names in the format {Class Name}::{Method Name} (for example, PDO::connect).

See https://github.com/google/re2/wiki/Syntax for syntax.

Monitoring rule conditions

Monitoring rule conditions specify the criteria for monitoring events. Monitoring rule conditions depend on the type of the rule, but must always contain the following members:

Monitoring rule conditions for rules of the types function-slow-exec, request-slow-exec, and request-high-mem-usage must contain the following additional member:

Monitoring rule conditions for the php-error type must contain the following additional member:

condition_id

The condition_id member is a positive integer value (zero included) uniquely identifying the condition within the rule.

severity

The severity member specifies the severity of the monitoring event generated when the condition is met.

threshold

The threshold member specifies the threshold at which the monitoring event with the given severity is generated.

For monitoring rules of the types function-slow-exec and request-slow-exec, the threshold value is a time in milliseconds.

For the request-high-mem-usage monitoring rule, the threshold value is a number of bytes in use at the end of the request.

mask

The mask member specifies a bitmask with PHP error types that generate the monitoring event with the given severity.

The mask can have the following values combined with a bitwise OR operation:

Monitoring rule actions

Monitoring rule actions specify actions that are executed when a monitoring event is generated.

Monitoring rule actions must contain the following members:

action_id

The action_id is a positive integer value (zero included) uniquely identifying the action within the monitoring rule.

action_type

The action_type member specifies the type of the action and must have one of the following values:

The url action performs an HTTP POST request with the monitoring event data to the given URL. See ZendHQ Monitoring Event Data for monitoring event data details.

action_target

The action_target member specifies the target of the action.

For "url" actions this is the url for the HTTP POST request.

ZendHQ Monitoring Event Data

If a monitoring rule contains monitoring event actions and the event is generated, then a JSON document with monitoring event data is included in the action.

The JSON document contains the following members:

event_id

The event_id member contains the ID of the monitoring event.

issue_id

The issue_id member contains the ID of the monitoring issue.

If event aggregation is enabled, then multiple monitoring events can have the same issue ID value.

request_id

The request_id member contains the ID of the request that caused the monitoring event.

If multiple monitoring events are generated during a request, they all have the same request_id value.

name

The name member contains the name of the monitoring rule (see rule_name) that triggered the event.

type

The type member contains the type of the monitoring rule (see rule_type) that triggered the event.

severity

The severity member is the severity of the monitoring event.

time_sec

The time_sec member contains the UNIX time of the event as a number of seconds since the EPOCH (January 1, 1970 midnight UTC/GMT).

request

The request member contains request details and has the following members:

function

The optional function member contains PHP function details and has the following members:

error

The optional error member contains PHP error details and has the following members:

stack_trace

The optional stack_trace member contains a JSON array with the PHP stack trace frames. Stack trace frames in the array contain the following members:

custom

The optional custom member contains custom event details and has the following members:

See Custom Monitoring Events for details.

variables

The optional variables member contains server variables and their values, such as the content of the $_SERVER variable. See PHP values for the details how PHP values are stored in JSON documents.

duration_sec

The optional duration_sec member contains the duration of a function or a request execution that triggered the event as a number of seconds.

memory_usage_bytes

The optional memory_usage_bytes member contains the number of used memory that triggered the event as a number of bytes.

Sample monitoring event data

{
    "event_id": 95,
    "issue_id": 95,
    "memory_usage_bytes": 49166992,
    "name": "High Memory Usage",
    "request": {
        "node_name": "node1",
        "php_version": "8.1.5",
        "pid": 68381,
        "url": "http://example.com/monitor.php?high_mem_usage"
    },
    "request_id": 321,
    "severity": "warning",
    "time_sec": 1652341693.624875,
    "type": "request-high-mem-usage",
    "variables": {
        "_REQUEST": {
            "high_mem_usage": ""
        },
        "_SERVER": {
            "DOCUMENT_ROOT": "/srv/http",
            "HTTP_ACCEPT": "text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8",
            "HTTP_ACCEPT_ENCODING": "gzip, deflate",
            "HTTP_ACCEPT_LANGUAGE": "en-GB,en;q=0.9",
            "HTTP_CONNECTION": "keep-alive",
            "HTTP_HOST": "example.com",
            "HTTP_REFERER": "http://example.com/monitor.php",
            "HTTP_UPGRADE_INSECURE_REQUESTS": "1",
            "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit
                                /605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15",
            "PHP_SELF": "/monitor.php",
            "QUERY_STRING": "high_mem_usage",
            "REMOTE_ADDR": "127.0.0.1",
            "REMOTE_PORT": "51914",
            "REQUEST_METHOD": "GET",
            "REQUEST_TIME": 1652341693,
            "REQUEST_TIME_FLOAT": 1652341693.463072,
            "REQUEST_URI": "/monitor.php?high_mem_usage",
            "SCRIPT_FILENAME": "/srv/http/monitor.php",
            "SCRIPT_NAME": "/monitor.php",
            "SERVER_NAME": "127.0.0.1",
            "SERVER_PORT": "80",
            "SERVER_PROTOCOL": "HTTP/1.1",
            "SERVER_SOFTWARE": "PHP 8.1.5 Development Server"
        }
    }
}

PHP values

The following rules are applied when PHP values are written into JSON documents:

Sample Monitoring Rules

Slow function execution rule

This rule triggers when the exec() PHP internal function or the slow_function() PHP user function takes too long to execute.

A critical monitoring event is generated if they take 1000 or more milliseconds and a warning monitoring event if they take 500 or more milliseconds to execute.

{
    "rule_id":  1,
    "rule_type":    "function-slow-exec",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "Slow Function Execution",
    "description":  "Triggered when one of the watched functions runs longer than the specified duration",
    "regex_match": "exec|slow_function",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 1000
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 500
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Function error rule

This rule triggers when the fopen() PHP internal function or the function_that_fails() PHP user function return false.

The severity of the monitoring event is a warning.

{
    "rule_id": 2,
    "rule_type": "function-error",
    "rule_parent_id": -1,
    "app_id": -1,
    "rule_name": "Function Error",
    "description": "Triggered when one of the watched functions returns FALSE",
    "regex_match": "fopen|function_that_fails",
    "conditions": [
        {
            "condition_id": 1,
            "severity": "warning"
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Slow request execution rule

This rule triggers when the PHP request takes too long to execute.

A critical monitoring event is generated if it takes 5000 or more milliseconds and a warning monitoring event if it takes 2000 or more milliseconds to execute.

{
    "rule_id":  3,
    "rule_type":    "request-slow-exec",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "Slow Request Execution",
    "description":  "Triggered when a PHP request's runtime is longer than the specified duration",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 5000
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 2000
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

High memory usage rule

This rule triggers when a PHP request uses too much memory.

A critical monitoring event is generated when 49152 or more bytes of memory is in use and a warning monitoring event is generated when 24576 or more bytes of memory is in use at the end of the request.

{
    "rule_id":  4,
    "rule_type":    "request-high-mem-usage",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "High Memory Usage",
    "description":  "Triggered when a PHP request consumes more than the specified amount of memory",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "threshold": 49152
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "threshold": 24576
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

PHP error rule

This rule triggers when one of the PHP errors specified in the conditions occurs.

A critical monitoring event is generated if the PHP error type matches the bitmask 0x0055 (E_ERROR, E_PARSE_ERROR, E_CORE_ERROR, and E_COMPILE_ERROR) and a warning monitoring event is generated if the PHP error type matches the bitmask 0x17A2 (E_WARNING, E_CORE_WARNING, E_COMPILE_WARNING, E_USER_ERROR, E_USER_WARNING, E_USER_NOTICE, E_RECOVERABLE_ERROR).

{
    "rule_id":  5,
    "rule_type":    "php-error",
    "rule_parent_id":   -1,
    "app_id":   -1,
    "rule_name":    "PHP Error",
    "description":  "Triggered when a PHP error of one of the selected error levels is reported",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "critical",
            "mask": 85
        },
        {
            "condition_id": 2,
            "severity": "warning",
            "mask": 6050
        }
    ],
    "actions": [
        {
            "action_id": 1,
            "action_type": "url",
            "action_target": "https://example.com/monitor.php"
        }
    ]
}

Custom event rule

This rule triggers when the monitor_custom_event() function or the monitor_custom_event_ex($ruleName = "Custom Event") function are called in the PHP script.

The severity of the monitoring event is a warning unless a specific value is given as the $userSeverity argument when calling the monitor_custom_event() function.

{
    "rule_id": 6,
    "rule_type": "custom",
    "rule_parent_id": -1,
    "rule_name" : "Custom Event",
    "description": "Triggered when the API function 'monitor_custom_event' is called",
    "conditions":
    [
        {
            "condition_id": 1,
            "severity": "warning"
        }
    ]
}