Skip to content

How to Test EMS Alarm Actions

Purpose

Prove the whole chain end to end: trigger an alarm in the simulator, confirm the EMS detects it, watch the EMS run the alarm's configured command sequence against the simulated device, then clear the alarm and confirm the protection reset behaves the way it was configured.

This is the guide to use when the question is "does the protection actually work?", as opposed to "does the register move?".

Prerequisites

  • Simulator stack up, target device served by the simulator.
  • ems-backend running with the alarm-action executor (alarm_actions.py) available.
  • The alarm's action list written — see HOW_TO_ADD_ALARM_ACTION.
  • The sequences the actions call must exist for that device's model.

How the pieces fit

Piece Where
Action list Inside the alarm map file, per alarm code
Alarm map files emsConfig/<category>/<prefix>_alarm_code.json (model variants add ___<model>)
Action editor EMS frontend, /alarm-actions-editor
Save endpoint POST /updateAlarmActions
Executor alarm_actions.py
Sequences emsConfig/<category>/sequences/<prefix>_command_sequences.json; PCS uses emsConfig/pcs/power_sequences/<MODEL>.json

Action fields the engine honours

Field Values Default
mode sequential, parallel sequential
failure_action abort, skip, retry, recovery_sequence abort
delay_ms milliseconds, per action 0
timeout_ms milliseconds, blank for none none
retries count 0
enabled true/false —
allow_on_clear must be exactly true for a clearance action to run absent

At most 32 actions per alarm (MAX_ACTIONS).

Two different failure vocabularies — do not mix them

Alarm actions use abort / skip / retry / recovery_sequence. PCS step-level sequencing uses its own, separate set: on_failure: abort | continue. Setting one where the other belongs fails validation.

Step-by-step procedure

Step 1 — Bring the stack up

Start ems-backend and the simulator. Confirm the target device is served: the Devices tile on the Alarms page should count it, and Triggerable should include it.

Step 2 — Select the device and identify its model

Use Active Models to confirm the device's manufacturer/model. The model decides which alarm map and which sequence file apply, so a wrong model here makes the rest of the test meaningless.

Step 3 — Inspect the alarm's actions

Open the alarm-action editor (/alarm-actions-editor) on the EMS frontend, choose the device and the alarm code, and check the action list. Record the expected sequence: order, target device, sequence name, mode.

Step 4 — Clear the write log

Simulator dashboard → Write Monitor → Clear log (POST /monitor/clear). This separates the commands you are about to cause from older traffic.

Step 5 — Trigger the alarm

Simulator dashboard → Alarms → find the device section → tick the rule → Trigger selected (or that row's Trigger).

Step 6 — Verify the simulator side

The register now holds a rule-violating value. The Firing now tile counts alarms as the EMS sees it — if that number rises, the EMS detected it.

Step 7 — Verify the EMS raised the alarm

Confirm in the EMS active alarm list that the alarm appears with the correct code and severity, then that it was recorded in TimescaleDB.

Step 8 — Verify the actions ran

Watch the Write Monitor. You should see the sequence's writes arrive, with:

  • the correct port for the target device
  • registers and values matching the sequence definition
  • timestamps in the expected order

Step 9 — Verify order, and sequential vs parallel

  • Sequential (mode: sequential): writes arrive in the defined order.
  • Parallel (mode: parallel): members of the same order group may interleave; each still waits its own delay_ms.

time has one-second resolution — enough to confirm ordering and gross delays, not sub-second timing.

Step 10 — Verify the device ended in the right state

Read the relevant registers back (HOW_TO_READ_WRITE_SIMULATOR_REGISTERS). A sequence that "ran" but left the device stopped is a failure.

Step 11 — Clear the alarm

Reset selected, or Reset all firing in the Active alarms panel.

Step 12 — Verify the reset behaviour

This is where AUTO and MANUAL differ — see below.

Testing AUTO vs MANUAL reset

A clearance action is one that runs on the cleared event. Whether it runs is decided by gate_reason, which requires all three:

  1. allow_on_clear is exactly true on the action,
  2. nothing else is still active on that device,
  3. the EMS is in an operation mode listed in allowed_modes.

Any one missing and the action does not run — the equipment is deliberately left as it is, because that is the recoverable mistake.

Configuration Behaviour How to test
allow_on_clear: true AUTO — clearance action runs once the alarm clears Clear the alarm, then watch the Write Monitor for the release sequence
allow_on_clear absent/false MANUAL — release stays latched Clear the alarm, confirm no release writes appear, then perform the authorised reset

To test the latch honestly, trigger two alarms on the same device, clear only one, and confirm the clearance action does not run while the second is still active — that exercises condition 2.

Watch the two statuses separately

Alarm state and protection latch status are independent. An alarm can clear while the protection remains latched. Do not read "the alarm list is empty" as "the protection has released".

Honest status of this feature

Claim Reality
The executor exists Yes — alarm_actions.py, with tests
Actions are per alarm code, per model Yes
A UI to edit them exists Yes — /alarm-actions-editor
The feature is in use in production No. Across the live emsConfig, 713 alarm-map files exist and not one carries an actions list

So a fresh install has an empty action list and triggering an alarm will run nothing. That is expected, not a fault: write the action first.

Verification

Check Expected
Alarm triggered Register holds the violating value
EMS detected Firing count rises; alarm in the EMS list
Persisted Row in TimescaleDB
Sequence loaded Correct manufacturer/model's sequence used
Target correct Writes land on the intended device's port
Order correct Sequential order holds; parallel groups interleave
Device responded Read-back shows the intended state
AUTO reset Release writes appear after clearance
MANUAL reset No release writes; latch persists until the authorised reset

Troubleshooting

Symptom Cause Fix
Triggered, EMS sees it, no writes The alarm has no actions list Add actions in /alarm-actions-editor
Writes go to the wrong device Action's device_key/model names another device Correct the action
Only the first action ran failure_action: abort and a later step failed Fix the failing step, or review the policy
Sequence name not found Wrong model, or sequence absent Confirm the model's sequence file
Clearance never runs allow_on_clear not exactly true, another alarm active, or the mode is not allowed Check all three gate conditions
Validation rejects a field Mixed the alarm-action and PCS step vocabularies Use abort/skip/retry/recovery_sequence here