The Activator system throttle prevents JVM node hard-crashes due to overload. The system throttle does not prevent processing, but rather limits the consumption of new work. The system throttle operates automatically. Throttle behavior is normally applied separately for each node of the cluster. You can also engage the throttle manually on all nodes of the cluster enable the system to finish processing all active messages. The occasional automatic engagement of a system throttle is good for Activator, and is a normal operating experience. When the system throttle engages, it does not mean that the JVM is not working at all, rather, it means that the JVM is working on tasks already in queue, until sufficient headroom is regained to accept new tasks.
The system throttle can be engaged for different reasons:
tuning.properties file setting).When a system throttle is engaged, consumption of new inbound work is halted. The trading engine stops polling and returns a 503 error to your trading partners when they try to connect.
Activator then re-prioritizes all traffic based upon these rules:
When the system throttle is invoked you may receive a log entry similar to the following (TaskScheduler load example):
| WARN [SystemThrottle] (SystemThrottle.run:213) - Engaging the SystemThrottle, TaskScheduler load is too large (limit = 150, current = 158) |
When system throttling is engaged, Message Tracker resend and reprocess attempts will fail. Resend and reprocess attempts will eventually generate "max attempt exceeded" errors.
| Note | System throttle settings should be changed only as part of a comprehensive tuning effort, best conducted with the help of Axway Professional Services. |
You manage general system throttle settings from the tuning.properties file. These settings are described in the following paragraphs. Additionally, you can manually engage/disengage system throttling at runtime in the UI. .
You manage the general system throttle behavior from the tuning.properties file, located in <install_directory>/conf.
The properties in this file are applied only to the node where the tuning.properties file is located. You must set the property for each node of a cluster by modifying the file for each node.
By default, tuning.properties is empty. This indicates that all of its possible entries are operating at their default values.
This chapter describes how to use the two properties related to system throttling that you can set in this file:
systemThrottle.pausePickups – Default=falsesystemThrottle.maximumTaskQueueSize – Default=150In some cases, you may want to force system throttling on startup. This is useful, for example, if you want to empty the system of messages waiting to be processed.
To to force startup in throttled mode:
<install_directory>/conf and open the tuning.properties file in a text editor.systemThrottle.pausePickups systemThrottle.pausePickups=true When the system restarts, it will operate in throttled mode. This status will be displayed for trading engine notes on the System Management page of the UI. You can temporarily override throttling by clicking Restart pickups for all trading engine nodes. When you restart Activator, it will restart in throttled mode until you modify this property.
By default, when Activator Task Scheduler load exceeds 150, the system throttle is engaged. You can modify this value. While there is no real limit to this value, there are practical limits due to environmental factors (physical memory size, server hardware, network speed, etc.).
We recommend that you change this property value in increments of 25.
To change the Task Scheduler load limit:
<install_directory>/conf and open the tuning.properties file in a text editor.systemThrottle.maximumTaskQueueSize systemThrottle.maximumTaskQueueSize=175 When the system restarts, it will engage throttling at the new load limit threshold.