Monitoring

During maintenance and monitoring each integration platform is very important to rely mainly on application logs, where there are a substantial and detailed information about running platform - OpenHub.

Application log

The main task for application log is hold the information from running of ESB which are required to solve any problems.

Sample of one logging line in following format:

%d{ISO8601} [${SERVERID}, %thread, %X{REQUEST_URI}, %X{REQUEST_ID}, %X{SESSION_ID}, %X{SOURCE_SYSTEM}, %X{CORRELATION_ID}, %X{PROCESS_ID}] %-5level %logger{36} : %msg%n

One log line contains standard parameters which can configure in config file of logging platform. In addition the following custom logging parameters are used:

LOG_SESSION_ID: unique HTTP session identifier. Thanks to this parameter can be grouped records from one user (system). This parameter is not filled if process is asynchronous and postponed (processed in the future).
REQUEST_URI: URI of incoming request to integration platform. This parameter is not filled if process is asynchronous and postponed (processed in the future).
REQUEST_ID: unique identifier of single request. Thanks to this parameter can grouped records from one request (HTTP request). This parameter is not filled if process is asynchronous and postponed (processed in the future).
SOURCE_SYSTEM: identifier of source system which send inbound request
CORRELATION_ID: unique identifier of single asynchronous message. Thanks to this parameter it can search all correlated records in log file for specific message. Note: this parameter is used by Log Viewer in admin gui to show correlated records.
PROCESS_ID: optional identifier of process which identifies records correlated to one business process e.g. creating new user which is composed by two subprocess: create new user and activate it.

Example:

2011-04-05 08:07:07,964 [server1, http-8080-Processor16, /sc-web-console/sc/sn/mock_demo_service/on/Demo, 127.0.1.1:4c8ed99e:12f244556e1:-7fdd, 6B6*****C6C2, deih3u36bh] INFO c.c.s.f.w.f.screen.ScreenFormPanel - the message for key 'widget16_label_key' was not found: form='demo_form1', locale='cs'

2013-08-07 02:01:34,542 [MACHINE_IS_UNDEFINED, Camel (camelContext) thread #25 - seda://asynch_message_route, , 192.168.198.100:-7d3dda4f:1405477688f:-60c4, , a2e7cf84-f4fe-e211-b400-005056bc0011] DEBUG

Substring in log line	Severity	Recommended action	Description of problem
Search string "] ERROR " in log files /srv/openhub/logs/j2ee/ (or where files are stored)	ERROR	Forward to resolve by support team	Fault status in the ESB application
Search string "] FATAL " in log files /srv/openhub/logs/j2ee/ (or where files are stored)	FATAL	Forward to resolve by support team	Critical status in the ESB application
Search string "was changed to FAILED"	WARN	Forward to resolving to administrator, who should look at where the problem is and why the message could not be processed.	Asynchronous message is in FAILED status (some business or technical error occurs)

Database

OpenHub stores in database records to support asynchronous processing of messages. If integration platform is processing synchronous request than information about that can be search in log file, no in database. Exception is logging and monitoring all request/responses sent by integration platform into external systems.

From monitoring of database of view is recommended to observe followings:

statuses of asynchronous messages in table message, column state (status) stavy - either directly by select into database or observe reports, see Admin GUI - Admin - Message report service.
when any information in message is changed, column last_update_timestamp is updated by actual time
statuses of external calls in table external_call, column state.

Detection	Severity	Recommended action	Description of problem
Message in FAILED state	WARN	Using the application log determine the causes of errors.	Asynchronous message ended up in the final status FAILED, which means that during the processing occurred an error (business or technical error).
Message is a long time in the state WAITING_FOR_RES	WARN	Using the application log to make sure that we sent a message to the external system properly and correctly and, if necessary, we even received an acknowledgment of receipt of our report from external system.	Asynchronous message is waiting to response from external system. If there are messages in this status for long time (normal response is received during a few of milliseconds) it means that the problem is on external system side.
Message is a long time in the state PROCESSING	WARN	Checks whether repair mechanisms works as expected and why messages are still in PROCESSING status.	Asynchronous message is in PROCESSING state. If some error occurs during processing a message and the message remains in this status, OpenHub has corrigible mechanism which changes after configurable time (default value is 300s) status of message to PARTLY_FAILED without changes of failed count (failed_count).
External call is in FAILED_END status	WARN	Using the application log determine the causes of errors.	Only external call - confirmations can be in this status and it means the problem where OpenHub could not confirm processing of asynchronous message. Confirmation of result of processing asynchronous messages is joined only with some external systems. If the confirmation fails a workflow process is stopped because external system can have some dependencies to next processing of it.
Message is a long time in the state POSTPONED	WARN	Check configurable parameters asynch.postponedInterval, which stands for after how long becomes inactive message in processing state (PROCESSING, WAITING, WAITING_FOR_RES) marked as "not processing".	Messages in status POSTPONED are processed with the same mechanism as messages in PARTLY_FAILED status. Only one difference is that postponed messages are processed preferably. Note: if the behavior of this mechanism would be unexpected (around PARTLY_FAILED messages processing) so it will be the same unexpected behaviour during POSTPONED processing.

See Operations which change message state for more details about state's workflow.