Overview

Mango runs every task at one of three levels, High, Medium and Low.  Some tasks have priorities that can be changed, others do not.  Mango provides a way to monitor the various rejections of tasks in the System Status --> Work Items view, this document describes what this information means and how to tune Mango for the best performance.

Task Queues for the High and Medium pools are used to ensure that a given task runs in the order it was scheduled.  The ordering is based on a unique identifier and the queue can be limited or infinite.  Limited queues can have a hard limit defined in the code or rely on the env property runtime.realTimeTimer.defaultTaskQueueSize.  It should be noted that only the High and Medium pools can run Ordered tasks, however they can also run tasks that do not require an order.  If an un-ordered task is submitted to the pool it will run at some point in the future but not necessarily before other tasks of the same type.

Understanding Runtime Performance

Information about the current status of running tasks and their rejections can be seen in the Work Items view under Administration > System Status > Work Items.  

Running Stats

This area provides insight into any tasks that are currently executing.  A task must be currently executing to be seen in this view, most tasks execute so quickly that this view will have little information in it in a healthy system.  When tasks start backing up and taking longer to run this page can provide insight into what tasks are long running and potentially degrading performance.

Rejected Stats

Information about task rejections can be found in this section. A task is rejected for one of 2 reasons.  First the queue may be full for an ordered task.  This means that the system cannot process the tasks fast enough and they have filled that task's specific queue and thus are being rejected.  This usually means that your timeouts are set improperly or the task is scheduled to execute too often.  Pool full rejections occur when the thread pool for that priority is full and no more task can be scheduled to run until a thread becomes free.

High Priority Tasks

High priority tasks run in the high priority pool at the highest thread priority of the Operating System and are ordered based on task.  For example tasks of different types may run at the same time while tasks such as the polling of a specific data source will run in its own queue such that the next poll can only happen after the current poll finishes.  Each high priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.

The High priority thread pool must be large enough to run all High priority tasks that are scheduled at any given time.  When the pool is full it will reject tasks with a Pool Full task rejection.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks, and the maximum pool size which allows limiting the pool so the system does not run out of resources when the pool is full.  When full, new tasks will be rejected until a thread becomes free to process new tasks.

Task Queue Size Ordered On Description
Point Value Batch Write 0 Batch Task Id Writes point values to database asychronously
Process Event Handler default yes execute system process of event handler
Set Point Value Unlimited yes set a point value asynchronously
Http Image Data Source 10 yes Retrieves images asynchronously
BACnet iAm Request N/A No ordering Sends iAm messages to devices
BACnet Object List Request N/A No ordering Requests object lists from devices
BACnet Point Creator N/A No ordering Create data points from discovery
BACnet COV Maintenance N/A No ordering Manage COV subscriptions
Brewer's Dashboard Delay Set yes 5 Set point values at some time in the future
Input stream epoll N/A No ordering Read input streams via epoll
Process epoll N/A No ordering Monitor and provide feedback on executed system processes
Persistent Point Hierarchy Sender N/A No ordering Synchronize point hierarchy
Persistent History Sync N/A No ordering Synchronized data point histories
Persistent Range Counter N/A No ordering Count a range of point values for a history sync
Persistent Sync Timer N/A No ordering Schedule the synchronization tasks
Persistent Timeout Output Stream N/A No ordering Check publisher's writing stream for disconnections
Interval Logging default Data Point XID Interval log point values
Polling Data Sources default Data Source XID Poll of each data source
User Event Cache Cleaner default Only 1 Purge memory of user events when user has not logged in for some time
Timeout Event Detectors default Event detector XID Any event detector that uses a timeout
Email Event Handler Escalation Task default Handler XID Send emails triggered by events
Nightly Purge 0 Only 1 Purge old data nightly
Upgrade Check 0 Only 1 Check for upgrades every 24hrs
Work Item Monitor 0 Only 1 Count executing work items every 10s
Schedule Configuration Backup 0 Only 1 Schedule a configuration backup task
Schedule Database Backup N/A No ordering Schedule a database backup task
Publisher send snapshot task default Publisher XID Send a snapshot of the publisher's information
Publisher send realtime data task default Publisher XID Send realtime data from publisher queue
License check task N/A No ordering Check for a valid license
Schedule Excel Report N/A No ordering Schedule a report to run now or later
JSON File Importer N/A No ordering Import a JSON Configuration via the JSON Import module
Maintenance Events 0 Maintenance Event XID Make event active or inactive
Schedule NoSQL Backup 0 Only 1 Schedule a NoSQL Backup to run now or later
Meta point delay N/A No ordering Delay the execution of a meta data point
Meta point cron N/A No ordering Execute meta point on CRON pattern
PakBus Network Poller N/A No ordering Periodically poll PakBus network for changes
Schedule Reports 5 Report XID Schedule reports to run now or later
Scheduled Event 0 Scheduled Event XID Make event active/inactive
Web Socket Ping Pong Tracker default session ID Manage websocket connectivity
NoSQL Data Map Cleaner 0 Only 1 Manage memory for writing point values
NoSQL Data Mover 0 Only 1 Collate point values for high performance writing
NoSQL Status Provider 0 Only 1 Provide status on write performance
PakBus Requests N/A No ordering Send requests on PakBus network
Persistent Data Source Status N/A No ordering Provide status on connections
Scripting Data Source Execution Delay N/A No ordering Delay execution of scripting data source
Serial Data Source timeout manager N/A No ordering Manage serial response timeouts
Advanced Schedule Events 1 Schedule XID Manage event state
Data Source Initializer N/A No ordering Initialize data sources when Mango starts
Data Source Terminator N/A No ordering Terminate data sources when Mango stops
Point Hierarchy Event Dispatcher N/A No ordering Dispatch events about the point hierarchy
Upgrade Downloader 0 Only 1 Download and install desired upgrades

Medium Priority Tasks

Medium priority tasks run in the medium priority pool just below the highest thread priority of the Operating System and are ordered based on task.  For example tasks of different types may run at the same time while tasks such as a system backup will run in its own queue such that the next backup can only happen after the current backup finishes.  Each medium priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.

The Medium priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work.  This pool is capable of queueing tasks if there are no free threads.  This means that in an overburdened system the Medium priority pool can grow unbounded.  When the pool is full it will queue tasks and run them when a thread becomes available.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.

Task Queue Size Ordered On Description
REST Temporary Resource Timeout default Resource ID Clean up expired resources
Data Point Event Unlimited Data Point XID and Listener ID Send messages about state and value changes on a point
Configuration Backup 0 Only 1 Backup JSON configuration
Database Backup default Only 1 Backup database
System setting changed notification 100 System Setting Update listeners as to what system setting was changed
Data File Import Status default resource ID REST temporary resource for data file imports
Haystack Import Status default resource ID REST temporary resource for history imports of haystack data
Snmp Mib Walk Task 0 Host and Port of Device Task to explore a device using an SNMP Mib file

Low Priority Tasks

Low priority tasks run in the low priority pool at the normal thread priority of the Operating System and are un-ordered. This means that any low priority task can run independent of any other low priority task.  The tasks are supplied to the pool and will be processed in no guaranteed order.  This pool behaves like the Medium priority pool such that tasks that  have no threads to process them are queued until a thread is free.

The Low priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work.  This pool is capable of queueing tasks if there are no free threads.  This means that in an overburdened system the Low priority pool can grow unbounded.  When the pool is full it will queue tasks and run them when a thread becomes available.  The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.

Task Description
Alarm notifications Notify about alarm state changes
Send Email Send email notifications
NoSQL Backup Backup NoSQL database(s)
NoSQL Restore Restore NoSQL database(s)

Configurable Priority

These tasks have a configurable priority via System Settings.  If run in the Medium or High pools they are ordered, if run in the Low pool the tasks have no order.

Task Queue Size Ordered On Description
Excel Report 10 Report XID Run an Excel Report
Report 5 Report XID Run a report from the Reports module