How to Test EMS Alarm Actions
Purpose
Prove the whole chain end to end: trigger an alarm in the simulator, confirm the EMS detects it, watch the EMS run the alarm's configured command sequence against the simulated device, then clear the alarm and confirm the protection reset behaves the way it was configured.
This is the guide to use when the question is "does the protection actually work?", as opposed to "does the register move?".
Prerequisites
- Simulator stack up, target device served by the simulator.
ems-backendrunning with the alarm-action executor (alarm_actions.py) available.- The alarm's action list written — see HOW_TO_ADD_ALARM_ACTION.
- The sequences the actions call must exist for that device's model.
How the pieces fit
| Piece | Where |
|---|---|
| Action list | Inside the alarm map file, per alarm code |
| Alarm map files | emsConfig/<category>/<prefix>_alarm_code.json (model variants add ___<model>) |
| Action editor | EMS frontend, /alarm-actions-editor |
| Save endpoint | POST /updateAlarmActions |
| Executor | alarm_actions.py |
| Sequences | emsConfig/<category>/sequences/<prefix>_command_sequences.json; PCS uses emsConfig/pcs/power_sequences/<MODEL>.json |
Action fields the engine honours
| Field | Values | Default |
|---|---|---|
mode |
sequential, parallel |
sequential |
failure_action |
abort, skip, retry, recovery_sequence |
abort |
delay_ms |
milliseconds, per action | 0 |
timeout_ms |
milliseconds, blank for none | none |
retries |
count | 0 |
enabled |
true/false | — |
allow_on_clear |
must be exactly true for a clearance action to run |
absent |
At most 32 actions per alarm (MAX_ACTIONS).
Two different failure vocabularies — do not mix them
Alarm actions use abort / skip / retry / recovery_sequence.
PCS step-level sequencing uses its own, separate set:
on_failure: abort | continue. Setting one where the other belongs fails
validation.
Step-by-step procedure
Step 1 — Bring the stack up
Start ems-backend and the simulator. Confirm the target device is served: the
Devices tile on the Alarms page should count it, and Triggerable should
include it.
Step 2 — Select the device and identify its model
Use Active Models to confirm the device's manufacturer/model. The model decides which alarm map and which sequence file apply, so a wrong model here makes the rest of the test meaningless.
Step 3 — Inspect the alarm's actions
Open the alarm-action editor (/alarm-actions-editor) on the EMS frontend, choose
the device and the alarm code, and check the action list. Record the expected
sequence: order, target device, sequence name, mode.
Step 4 — Clear the write log
Simulator dashboard → Write Monitor → Clear log (POST /monitor/clear).
This separates the commands you are about to cause from older traffic.
Step 5 — Trigger the alarm
Simulator dashboard → Alarms → find the device section → tick the rule → Trigger selected (or that row's Trigger).
Step 6 — Verify the simulator side
The register now holds a rule-violating value. The Firing now tile counts alarms as the EMS sees it — if that number rises, the EMS detected it.
Step 7 — Verify the EMS raised the alarm
Confirm in the EMS active alarm list that the alarm appears with the correct code and severity, then that it was recorded in TimescaleDB.
Step 8 — Verify the actions ran
Watch the Write Monitor. You should see the sequence's writes arrive, with:
- the correct port for the target device
- registers and values matching the sequence definition
- timestamps in the expected order
Step 9 — Verify order, and sequential vs parallel
- Sequential (
mode: sequential): writes arrive in the defined order. - Parallel (
mode: parallel): members of the same order group may interleave; each still waits its owndelay_ms.
time has one-second resolution — enough to confirm ordering and gross delays,
not sub-second timing.
Step 10 — Verify the device ended in the right state
Read the relevant registers back (HOW_TO_READ_WRITE_SIMULATOR_REGISTERS). A sequence that "ran" but left the device stopped is a failure.
Step 11 — Clear the alarm
Reset selected, or Reset all firing in the Active alarms panel.
Step 12 — Verify the reset behaviour
This is where AUTO and MANUAL differ — see below.
Testing AUTO vs MANUAL reset
A clearance action is one that runs on the cleared event. Whether it runs is
decided by gate_reason, which requires all three:
allow_on_clearis exactlytrueon the action,- nothing else is still active on that device,
- the EMS is in an operation mode listed in
allowed_modes.
Any one missing and the action does not run — the equipment is deliberately left as it is, because that is the recoverable mistake.
| Configuration | Behaviour | How to test |
|---|---|---|
allow_on_clear: true |
AUTO — clearance action runs once the alarm clears | Clear the alarm, then watch the Write Monitor for the release sequence |
allow_on_clear absent/false |
MANUAL — release stays latched | Clear the alarm, confirm no release writes appear, then perform the authorised reset |
To test the latch honestly, trigger two alarms on the same device, clear only one, and confirm the clearance action does not run while the second is still active — that exercises condition 2.
Watch the two statuses separately
Alarm state and protection latch status are independent. An alarm can clear while the protection remains latched. Do not read "the alarm list is empty" as "the protection has released".
Honest status of this feature
| Claim | Reality |
|---|---|
| The executor exists | Yes — alarm_actions.py, with tests |
| Actions are per alarm code, per model | Yes |
| A UI to edit them exists | Yes — /alarm-actions-editor |
| The feature is in use in production | No. Across the live emsConfig, 713 alarm-map files exist and not one carries an actions list |
So a fresh install has an empty action list and triggering an alarm will run nothing. That is expected, not a fault: write the action first.
Verification
| Check | Expected |
|---|---|
| Alarm triggered | Register holds the violating value |
| EMS detected | Firing count rises; alarm in the EMS list |
| Persisted | Row in TimescaleDB |
| Sequence loaded | Correct manufacturer/model's sequence used |
| Target correct | Writes land on the intended device's port |
| Order correct | Sequential order holds; parallel groups interleave |
| Device responded | Read-back shows the intended state |
| AUTO reset | Release writes appear after clearance |
| MANUAL reset | No release writes; latch persists until the authorised reset |
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Triggered, EMS sees it, no writes | The alarm has no actions list | Add actions in /alarm-actions-editor |
| Writes go to the wrong device | Action's device_key/model names another device |
Correct the action |
| Only the first action ran | failure_action: abort and a later step failed |
Fix the failing step, or review the policy |
| Sequence name not found | Wrong model, or sequence absent | Confirm the model's sequence file |
| Clearance never runs | allow_on_clear not exactly true, another alarm active, or the mode is not allowed |
Check all three gate conditions |
| Validation rejects a field | Mixed the alarm-action and PCS step vocabularies | Use abort/skip/retry/recovery_sequence here |