Alerts

Created by Trello
Last updated: Nov 08, 2017 by Petr Jůza

Alerts define metrics for watching database data and if any metric exceeds its limit then alert is activated and further operation can be executed.

Metrics are configured - the SQL query for getting count of items and limit for checking.

Examples of alerts (also with follow-up operations):

  • when count of failed messages for last 10 minutes exceeds 5 then sends email to administrators
  • when count of messages which wait for response from external system for more then 5 minutes exceeds 10 then sends email to administrators

Alerts checking is scheduled operation that is determined by ohf.alerts.repeatTime configuration parameter - checking is executed every 5 minutes by default.

Alerts configuration

There are the following configuration possibilites:

  • property files (default option)
  • JMX

Properties alerts configuration

Alerts configuration uses the same model as rest of OpenHub framework, see External configuration model 

Property file configuration format and example (from application.properties):

###############################################################################
#  Alerts configuration
#
#   There the following property names:
#   - alerts.N.id: unique alert identification (if not defined then order number (=N) is used instead)
#   - alerts.N.limit: limit that must be exceeded to activate alert
#   - alerts.N.sql: SQL query that returns count of items for comparison with limit value
#   - [alerts.N.enabled]: if specified alert is enabled or disabled; enabled is by default
#   - [alerts.N.mail.subject]: notification (email, sms) subject; can be used Java Formatter placeholders (%s = alert ID)
#   - [alerts.N.mail.body]: notification (email, sms) body; can be used Java Formatter placeholders (%d = actual count, %d = limit)
#
###############################################################################

# checks if there is any waiting message that exceeds time limit for timeout
alerts.900.id=WAITING_MSG_ALERT
alerts.900.limit=0
alerts.900.sql=SELECT COUNT(*) FROM message WHERE state = 'WAITING_FOR_RES' AND last_update_timestamp < (current_timestamp - interval '3600 seconds')

Alerts from core module start from 900, project-specific alerts start from 0. This order number is only for better description, doesn't have influence to behaviour.

SQL queries must be defined for specific database that is use.

JMX alerts configuration

JMX configuration allows change alert limits and enable/disable selected alerts.

Reaction to alert activation

There are listeners org.openhubframework.openhub.spi.alerts.AlertListener which are called when specified alert is activated. 

There is default org.openhubframework.openhub.core.alerts.EmailAlertListenerSupport implementation that  sends email notifications to admin emails.

If you want to implement more actions and you can implement AlertListener or extends default implementation EmailAlertListenerSupport.