During maintenance and monitoring each integration platform is very important to rely mainly on application logs, where there are a substantial and detailed information about running platform - OpenHub.

Application log

The main task for application log is hold the information from running of ESB which are required to solve any problems.

Sample of one logging line in following format:

%d{ISO8601} [${SERVERID}, %thread, %X{REQUEST_URI}, %X{REQUEST_ID}, %X{SESSION_ID}, %X{SOURCE_SYSTEM}, %X{CORRELATION_ID}, %X{PROCESS_ID}] %-5level %logger{36} : %msg%n

One log line contains standard parameters which can configure in config file of logging platform. In addition the following custom logging parameters are used:

Example:

2011-04-05 08:07:07,964 [server1, http-8080-Processor16, /sc-web-console/sc/sn/mock_demo_service/on/Demo, 127.0.1.1:4c8ed99e:12f244556e1:-7fdd, 6B6*****C6C2, deih3u36bh] INFO  c.c.s.f.w.f.screen.ScreenFormPanel - the message for key 'widget16_label_key' was not found: form='demo_form1', locale='cs'

2013-08-07 02:01:34,542 [MACHINE_IS_UNDEFINED, Camel (camelContext) thread #25 - seda://asynch_message_route, , 192.168.198.100:-7d3dda4f:1405477688f:-60c4, , a2e7cf84-f4fe-e211-b400-005056bc0011] DEBUG

 

Substring in log line

Severity

Recommended action

Description of problem

Search string "] ERROR " in log files /srv/openhub/logs/j2ee/ (or where files are stored)

 

ERROR

Forward to resolve by support team

 

Fault status in the ESB application

Search string "] FATAL " in log files /srv/openhub/logs/j2ee/ (or where files are stored)

FATAL

Forward to resolve by support team

 

Critical status in the ESB application

Search string "was changed to FAILED"

WARN

Forward to resolving to administrator, who should look at where the problem is and why the message could not be processed.

Asynchronous message is in FAILED status (some business or technical error occurs)

Database

OpenHub stores in database records to support asynchronous processing of messages. If integration platform is processing synchronous request than information about that can be search in log file, no in database. Exception is logging and monitoring all request/responses sent by integration platform into external systems.

From monitoring of database of view is recommended to observe followings:

 

Detection

Severity

Recommended action

Description of problem

Message in FAILED state

 

WARN

Using the application log determine the causes of errors.

Asynchronous message ended up in the final status FAILED, which means that during the processing occurred an error (business or technical error).

Message is a long time in the state WAITING_FOR_RES

WARN

Using the application log to make sure that we sent a message to the external system properly and correctly and, if necessary, we even received an acknowledgment of receipt of our report from external system.

 

Asynchronous message is waiting to response from external system.

If there are messages in this status for long time (normal response is received during a few of milliseconds) it means that the problem is on external system side.

Message is a long time in the state PROCESSING

WARN

Checks whether repair mechanisms works as expected and why messages are still in PROCESSING status.

 

Asynchronous message is in PROCESSING state.

If some error occurs during processing a message and the message remains in this status, OpenHub has corrigible mechanism which changes after configurable  time (default value is 300s) status of message to PARTLY_FAILED without changes of failed count (failed_count).

External call is in FAILED_END status

WARN

Using the application log determine the causes of errors.

Only external call - confirmations can be in this status and it means the problem where OpenHub could not confirm processing of asynchronous message.

Confirmation of result of processing asynchronous messages is joined only with some external systems. If the confirmation fails a workflow process is stopped because external system can have some dependencies to next processing of it.

Message is a long time in the state POSTPONEDWARNCheck configurable parameters asynch.postponedInterval, which stands for after how long becomes inactive message in processing state (PROCESSING, WAITING, WAITING_FOR_RES) marked as "not processing".

Messages in status POSTPONED are processed with the same mechanism as messages in PARTLY_FAILED status. Only one difference is that postponed messages are processed preferably.

Note: if the behavior of this mechanism would be unexpected (around PARTLY_FAILED messages processing) so it will be the same unexpected behaviour during POSTPONED processing.


See Operations which change message state for more details about state's workflow.