How to collect statistics on a BusinessWorks or BusinessWorks Container Edition configuration using the Smart Engine

One challenge commonly faced when working on high cpu usage, high memory usage or poor performances issues is that there is need to collect data at OS, JVM and BusinessWorks levels at the same time and to create an aggregated view of all those elements.

This is generally difficult to do while different tools like tops, jstack (for thread dumps), JMX / JVisualVM and the TEA must be used for this.

To make things simple and easy a feature called the Smart Engine has been introduced in BusinessWorks 6.8 and BusinessWorks Container Edition 2.7.

The Smart Engine is not a monitoring or observability solution active permanently, this is a tool available on demand to help analyze issues where there is need to have a global view on the behavior of a configuration. It can be used also while executing load or performance tests. When creating a support case you may also use Smart Engine to generate a report with all the details on your environment and attach it to the case.

This new article is to explain how to use the Smart Engine.

# Enabling the Smart Engine

This can be done simply by calling the following API on the appnode OSGi endpoint:

GET http://<host>:<port>/monitor/systemproperties/enableSmartEngine?bw.smartengine.appStatistics.enabled=true

Enabling the Smart Engine from Postman

The <host> is the name of the machine hosting the target appnode, the <port> is the OSGi port of the target appnode (defined by the property org.osgi.service.http.port in the appnode config.ini file).

The query parameter ‘bw.smartengine.appStatistics.enabled=true’ is optional, and not enabled by default, I recommend enabling it while this adds process and activity statistics in the reports. This is equivalent to enabling Process Instrumentation using the TEA, bwadmin or the bwagent API.

Obviously enabling the Smart Engine and the process instrumentation will induce a little overhead on the engine but this is generally almost neglectable.

Once the Smart Engine is enabled, if there is already some load you can just wait for data to accumulate for a few minutes or you can start generating load if you are doing tests.

Tip: before enabling the SmartEngine you may want to archive or delete reports that may exist from a previous test session. Deleting reports can be done by stopping the appnode and deleting the files in the report directory or by using the following API:

GET http:// <host>:<port>/monitor/reports/deleteall

# Generating a report

You can generate a report by calling the following API:

http://<host>:<port>/monitor/reports/generate

Generating a report from Postman

By default, reports are written, in zip format, in the <TIBCO_HOME>/bw/6.X/reports directory but this is configurable with a property.

You can see below an example of a reports directory content with three reports in it:

Content of a sample reports directory

By default five reports are kept in the reports directory, this can be changed with a property.

# Reviewing a report

To review a report you need first to unzip the generated file, then you just have to open the html file in a browser.

Example of a report zip file content

A report is made of different sections that are accessible via the links available on the top of it:

The report top menu

For example, you can access to the JVM details by clicking on the ‘JVM Information’ link.

The sections available are listed below. Details about each section are available in the SmartEngine documentation:

https://docs.tibco.com/pub/activematrix_businessworks/6.11.0/doc/html/Default.htm#administration/Generating_Reports_for_Engine_Data.htm?TocPath=Administration%257CSmart%2520Engine%257CGenerating%2520Reports%2520for%2520Engine%2520Data%257C_____0

BW Application

This section provides details about the BusinessWorks applications, processes and activities execution. It is populated only if Process Instrumentation is available or if Smart Engine has been enabled with the bw.smartengine.appStatistics.enabled=true parameter.

Example of the BWApplication section

Operating System
This section provides details about the Operating System layer including the cpu load, memory usage and swap space usage.

Example of the Operating System section

Runtime Information

This section provides details about the runtime environment of the appnode, including environment variables and Java properties.

Example of the Operating System section

JVM Information

This section provides details about the JVM including the Java version and elements about the JVM behavior.

Example of the JVM Information section

Memory Information

This section provides details about the JVM memory usage including the actual Heap Size and used Heap Size.

Example of the JVM Information section

Thread Information

This section provides details about the JVM threads including the number of created and active threads.

Example of the Thread Information section

Thread List

This section provides the list of active JVM threads.

Example of the Thread List section

Thread Dump

This section provides a dump for all the active JVM threads.

Class Loading

This section provides details about loaded classes.

Objects SnapShot

This section provides details about loaded classes heap usage.

Example of the Object Snapshot section

Analysis

This section is filled only if a trigger occurs.

Recommendations
This section is filled only if a trigger occurs.

# Managing reports

You can manage available reports by using the Smart Engine reports API, details are available in the documentation:

https://docs.tibco.com/pub/activematrix_businessworks/6.11.0/doc/html/Default.htm#administration/Reports_REST_API.htm?TocPath=Administration%257CSmart%2520Engine%257CGenerating%2520Reports%2520for%2520Engine%2520Data%257C_____3

# Triggers

The Smart Engine manages a number of predefined triggers that will automatically generate a report when certain conditions are met, in such cases elements are included in the Analysis and Recommendations sections of the generated report.

For example, there is a trigger that monitors Heap size usage, it will occur when Heap size usage if above 80 % of max Heap size for more than 5 minutes.

In such case the Analysis and Recommendations sections of the report are filled with the elements below:

Example of the Analysis and Recommendations sections

The list of predefined triggers is included at the end of this article.

It is possible to update or delete existing triggers and create new triggers using a REST API. The documentation is available here:

https://docs.tibco.com/pub/activematrix_businessworks/6.11.0/doc/html/Default.htm#administration/Triggers_REST_API.htm?TocPath=Administration%257CSmart%2520Engine%257CGenerating%2520Reports%2520for%2520Engine%2520Data%257C_____2

# Disabling the SmartEngine

This can be done simply by calling the following API on the appnode OSGi endpoint:

GET http://<host>:<port>/monitor/systemproperties/disableSmartEngine?bw.smartengine.appStatistics.enabled=false

# Available properties

The following properties can be added in the target appnode config.ini file to configure the SmartEngine.

This property can be set to enable the SmartEngine at appnode start-up (by default it is disabled):
bw.smartengine.enabled=true

This property can be set to keep recent reports (by default it is enabled):
bw.smartengine.keepRecentReports.enabled=true

This property can be set to set the number of reports that are kept by the Smart Engine (by default this is 5):
bw.smartengine.keepRecentReports.size=10

# Reference elements

The SmartEngine documentation is available at the following URL:

https://docs.tibco.com/pub/activematrix_businessworks/6.11.0/doc/html/Default.htm#administration/smart-engine.htm?TocPath=Administration%257CSmart%2520Engine%257C_____0

# Additional elements

The list of triggers that are predefined in BusinessWorks 6.10 is available below:

[
    {
        "id": "bw.frwk.trigger.HighAppsTotalActiveToCompletedJobsRatioTriggerAction",
        "name": "High Apps Total Active-to-Completed Jobs Ratio Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighAppsTotalActiveToCompletedJobsRatioTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highTotalActiveToCompletedJobsRatioThresholdPercent": 200.0,
            "highTotalActiveToCompletedJobsRatioDurationMins": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.HighProcessesTotalActiveToCompletedJobsRatioTriggerAction",
        "name": "High Processes Total Active-to-Completed Jobs Ratio Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighProcessesTotalActiveToCompletedJobsRatioTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highTotalActiveToCompletedJobsRatioThresholdPercent": 200.0,
            "highTotalActiveToCompletedJobsRatioDurationMins": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.HighAppsNewActiveToCompletedJobsRatioTriggerAction",
        "name": "High Apps New Active-to-Completed Jobs Ratio Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighAppsNewActiveToCompletedJobsRatioTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highNewlyActiveToCompletedJobsRatioThresholdPercent": 200.0,
            "highNewlyActiveToCompletedJobsRatioDurationMins": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.HighProcessesNewActiveToCompletedJobsRatioTriggerAction",
        "name": "High Processes New Active-to-Completed Jobs Ratio Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighProcessesNewActiveToCompletedJobsRatioTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highNewlyActiveToCompletedJobsRatioThresholdPercent": 200.0,
            "highNewlyActiveToCompletedJobsRatioDurationMins": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.HighCPUWithGovernanceSecurityTriggerAction",
        "name": "High CPU with Governance Security Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighCPUWithGovernanceSecurityTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highCpuThresholdPercent": 80.0,
            "highCpuDurationMins": 5.0,
            "methodTotalThreadsCpuTimeThresholdMins": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.ActivityExceptionTriggerAction",
        "name": "Activity Exception Trigger",
        "enabled": false,
        "properties": {
            "actionId": "ActivityExceptionTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false
        }
    },
    {
        "id": "bw.frwk.trigger.HighActivityWaitingThreadsTriggerAction",
        "name": "High Activity Waiting Threads Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighActivityWaitingThreadsTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highActivityWaitingThreadsCountThreshold": 200,
            "highActivityWaitingThreadsDurationMinutesThreshold": 5.0
        }
    },
    {
        "id": "bw.frwk.trigger.FlowLimitAutoTuneTriggerAction$0",
        "name": null,
        "enabled": false,
        "properties": {
            "actionId": "FlowLimitAutoTuneTriggerAction",
            "generateReport": false,
            "generateEngineData": false,
            "updateSystemProperty": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "autoTune": false,
            "autoTuneComponents": "",
            "autoDecreaseCapacity": true,
            "autoIncreaseCapacity": true,
            "capacityMargin": 1,
            "deltaCapacityByPercent": true,
            "deltaCapacityByValue": false,
            "deltaCapacityPercent": 50.0,
            "deltaCapacityValue": 20,
            "deltaMargin": 1,
            "highOccupancyRatioThresholdPercent": 100.0,
            "lowOccupancyRatioThresholdPercent": 20.0,
            "maxCapacity": 50,
            "minCapacity": 5
        }
    },
    {
        "id": "bw.frwk.trigger.FlowLimitAutoTuneTriggerAction$1",
        "name": null,
        "enabled": false,
        "properties": {
            "actionId": "FlowLimitAutoTuneTriggerAction",
            "generateReport": false,
            "generateEngineData": false,
            "updateSystemProperty": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "autoTune": false,
            "autoTuneComponents": "",
            "autoDecreaseCapacity": true,
            "autoIncreaseCapacity": true,
            "capacityMargin": 1,
            "deltaCapacityByPercent": true,
            "deltaCapacityByValue": false,
            "deltaCapacityPercent": 50.0,
            "deltaCapacityValue": 50,
            "deltaMargin": 1,
            "highOccupancyRatioThresholdPercent": 100.0,
            "lowOccupancyRatioThresholdPercent": 20.0,
            "maxCapacity": 100,
            "minCapacity": 10
        }
    },
    {
        "id": "bw.frwk.trigger.FlowLimitAutoTuneTriggerAction$2",
        "name": null,
        "enabled": false,
        "properties": {
            "actionId": "FlowLimitAutoTuneTriggerAction",
            "generateReport": false,
            "generateEngineData": false,
            "updateSystemProperty": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "autoTune": false,
            "autoTuneComponents": "",
            "autoDecreaseCapacity": true,
            "autoIncreaseCapacity": true,
            "capacityMargin": 1,
            "deltaCapacityByPercent": true,
            "deltaCapacityByValue": false,
            "deltaCapacityPercent": 50.0,
            "deltaCapacityValue": 100,
            "deltaMargin": 1,
            "highOccupancyRatioThresholdPercent": 100.0,
            "lowOccupancyRatioThresholdPercent": 20.0,
            "maxCapacity": 1000,
            "minCapacity": 100
        }
    },
    {
        "id": "bw.montr.trigger.HighCPUTrigger",
        "name": "High CPU Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighCPUTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highCpuThresholdPercent": 80.0,
            "highCpuDurationMins": 5.0
        }
    },
    {
        "id": "bw.montr.trigger.HighMemoryTrigger",
        "name": "High Memory Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighMemoryTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highMemoryThresholdPercent": 80.0,
            "highMemoryDurationMins": 5.0
        }
    },
    {
        "id": "bw.montr.trigger.OutOfMemoryTrigger",
        "name": "Out of Memory Trigger",
        "enabled": false,
        "properties": {
            "actionId": "OutOfMemoryTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "outOfMemoryThresholdPercent": 95.0
        }
    },
    {
        "id": "bw.montr.trigger.HighLiveThreadsTrigger",
        "name": "High Live Threads Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighLiveThreadsTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highLiveThreadsThreshold": 500,
            "highLiveThreadsDurationMins": 5.0
        }
    },
    {
        "id": "bw.montr.trigger.HighBlockedThreadsTriggerAction",
        "name": "High Blocked Threads Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighBlockedThreadsTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highBlockedThreadsThreshold": 15,
            "highBlockedThreadsDurationMins": 5.0
        }
    },
    {
        "id": "bw.sharedresource.trigger.HighAppsHttpConnectorIdleTimeoutTriggerAction",
        "name": "High App HTTP Connector Idle Timeout Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighAppsHttpConnectorIdleTimeoutTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "highIdleTimeoutPerMinuteThreshold": 60,
            "highIdleTimeoutDurationMinutesThreshold": 5.0
        }
    },
    {
        "id": "bw.sharedresource.trigger.HttpConnectorAcceptorThreadCountThresholdTriggerAction",
        "name": "HTTP Connector Acceptor Count Threshold trigger",
        "enabled": false,
        "properties": {
            "actionId": "HttpConnectorAcceptorThreadCountThresholdTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false
        }
    },
    {
        "id": "bw.sharedresource.trigger.HttpConnectorExecutorThreadpoolUtilizationThresholdTriggerAction",
        "name": "HTTP Connector threadpool utilization Threshold trigger",
        "enabled": false,
        "properties": {
            "actionId": "HttpConnectorExecutorThreadpoolUtilizationThresholdTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "executorThreadpoolUtilizationThreshold": 85.0,
            "highThreadpoolUtilizationDurationMinutesThreshold": 5.0
        }
    },
    {
        "id": "bw.sharedresource.trigger.HttpConnectorQueueUtilizationThresholdTriggerAction",
        "name": "HTTP Connector Blocking Queue utilization Threshold trigger",
        "enabled": false,
        "properties": {
            "actionId": "HttpConnectorQueueUtilizationThresholdTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "connectorThreadpoolQueueUtilizationThreshold": 85.0
        }
    },
    {
        "id": "bw.sharedresource.trigger.HighQueuePendingMessagesTrendingTriggerAction",
        "name": "High JMS Queue Pending Messages Trending Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighQueuePendingMessagesTrendingTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "queuePendingMessagesCountMinValueThreshold": 1000,
            "queuePendingMessagesTrendingPercentThreshold": 300,
            "queuePendingMessagesDurationMinutesThreshold": 5.0
        }
    },
    {
        "id": "bw.sharedresource.trigger.HighQueuePendingMessagesCountTriggerAction",
        "name": "High JMS Queue Pending Messages Count Trigger",
        "enabled": false,
        "properties": {
            "actionId": "HighQueuePendingMessagesCountTriggerAction",
            "generateReport": true,
            "generateEngineData": true,
            "enabled": true,
            "debug": false,
            "test": false,
            "queuePendingMessagesCountThreshold": 10000,
            "queuePendingMessagesDurationMinutesThreshold": 5.0
        }
    }
]