How to Test Device Command Sequences
Verify a command sequence against the BESS simulator before it ever touches real equipment, and understand exactly what the code does and does not let you do on live plant.
Purpose
Prove that a sequence does what its author intended: the right registers, the right encodings, the right values, and the right outcome when a step fails. The test harness is built for this and enforces simulator-first operation in code.
Prerequisites
- The EMS backend running, with
pyModbusTCPavailable. - A device whose model defines the sequence under test — confirm with
--device <engine-key> --list-actions. - For simulator testing: a Modbus endpoint whose host is recognisably a simulation.
- For physical testing: the two gates below, satisfied deliberately.
Relevant Files
| Path | Role |
|---|---|
command_sequence_test.py |
The harness: run_test, start_test, stop_test |
device_command_sequences.py |
The engine and its CLI |
ems_settings_api.py |
The REST test endpoints |
protocol_adapters.py |
The adapters the test drives |
resources/views/ |
The command-sequence editor and execution screens |
A. Simulator Testing
What counts as a simulator
The harness decides from the host name, and it is deliberately generous about what looks like a simulation:
_SIMULATOR_HINTS = ("simulator", "simulated", "sim-backend", "mock", "fake")
def is_simulator_host(host) -> bool:
text = str(host or "").strip().lower()
if not text:
return False
if text in _LOOPBACK:
return True
return any(hint in text for hint in _SIMULATOR_HINTS)
Loopback addresses always count. Otherwise the host string must contain one of those hints.
The resolved endpoint reports this as "simulator": true|false, and the endpoint is read from
the same configuration node the EMS itself polls — "no second, drifting notion of the
device address." So a test targets exactly what the EMS is driving, not a separate address
someone typed in.
Steps
- Start the required services. The backend must be up, and the simulator must be listening on the device's configured host and port.
- Identify the device's endpoint — where it is and whether the harness considers it a simulation:
python3 -c "from command_sequence_test import resolve_endpoint; import json; \
print(json.dumps(resolve_endpoint('<device_type>', '<device_key>'), indent=2))"
Check simulator and source in the output. source names the exact config path the
endpoint came from, which is how you prove you are testing what you think.
- Confirm the model and its sequences:
python3 device_command_sequences.py --device <engine-key> --list-models
python3 device_command_sequences.py --device <engine-key> --model <MODEL> --list-actions
- Look at the steps before running them.
configured_steps(device_type, model, action)returns the sequence's steps as the engine will execute them, so a wrong address is visible before a single write:
python3 -c "from command_sequence_test import configured_steps; import json; \
print(json.dumps(configured_steps('<category>','<MODEL>','<action>'), indent=2))"
- Open the command sequence editor and select the target device.
- Select the sequence — the picker lists the model's real
actionskeys. - Enter the dynamic input parameters. Each
${name}placeholder in the steps must be supplied; an unsupplied placeholder cannot resolve. - Validate without executing — this writes nothing:
POST /api/v1/ems/command-sequences/validatewithdevice_type,modelanddata. - Execute against the simulator.
-
Monitor each step. Steps report one of:
_STEP_RESULT = {"ok": "PASS", "command": "PASS", "skipped": "SKIPPED", "failed": "FAIL", "timeout": "ERROR", "error": "ERROR"} -
Verify register writes and responses. A step with
verify: truereads the register back; the measured value is reported alongside the expected one. - Check delays and conditions took effect in order.
- Review the logs, and the
--debugoutput if a step is unexpected. - Read the verdict. The report's
statusisPASSorFAIL, with elapsed time stamped.FAILwhen any row isFAILorERROR.
The default path is the safe one
Nothing above needs a flag. The harness refuses a non-simulator host unless the caller
explicitly passes authorize_physical=True, so the default path is the simulator path.
B. Testing from Python
from command_sequence_test import resolve_endpoint, run_test, start_test, stop_test
# 1. Show where this will run, and whether it is a simulation.
endpoint = resolve_endpoint("ev_charger", "EV-1")
print(endpoint["host"], endpoint["port"], "simulator:", endpoint["simulator"])
# 2. Synchronous run against the simulator.
report = run_test("ev_charger", "set_current_limit",
device_key="EV-1",
parameters={"current_a": 16})
print("status:", report["status"]) # PASS / FAIL / SKIPPED / STOPPED
for row in report["steps"]: # the per-step rows live under "steps"
print(" ", row["status"], row["step"], row["actual"])
# 3. Asynchronous run, for a sequence long enough to want to watch it.
job = start_test("ev_charger", "set_current_limit",
device_key="EV-1", parameters={"current_a": 16})
print("job id:", job["test_id"]) # the job key is "test_id", not "id"
# 4. Cancel it.
print(stop_test(job["test_id"]))
start_test is the asynchronous entry point and returns a job record whose test_id is what
stop_test takes. That pairing is the supported way to cancel — there is no cancellation on
run_action, so a long sequence you may need to abort should go through the test path. A
cancelled run comes back with status: "STOPPED" and the steps that never ran marked
SKIPPED.
Errors you will see
| Exception | Meaning |
|---|---|
SequenceTestError |
Base class for the three below — the test could not be set up, often no endpoint configured |
TestNotFound |
No such test id |
TestConflict |
A conflicting test is already running on that device |
SequenceStopped |
The test was cancelled. Imported from pcs_power_sequence, not from command_sequence_test |
A device with no usable endpoint raises
<label> '<device_key>' has no endpoint configured.
C. Live Device Testing
Physical execution is gated by two independent checks, both of which must be satisfied:
if target == "physical" and not authorize_physical:
# refused
if not endpoint["simulator"] and not authorize_physical:
# refused: "...because it is not a simulator. Pass authorize_physical to ..."
The module's own reasoning: "asking for physical execution is never enough on its own."
Declaring target="physical" does not authorise anything — the flag is required separately,
and a host that merely fails to look like a simulator still needs it. A typo in a host name
cannot accidentally reach live equipment.
Before a physical run, confirm each of these:
- User authorization.
authorize_physical=Truepassed deliberately by a person who intends it. - EMS operational mode. The mode gates clearance actions through
gate_reason'sallowed_modes; a mode that forbids clearance actions will refuse them. - Device communication. The endpoint resolves and the TCP connection opens; otherwise
cannot open Modbus connection to host:port. - Safety interlocks. Whatever the site relies on; the harness does not substitute for them.
- Register and parameter validation. ⚠️ There is none. A step's
addressis not cross-checked against the model's register map, and a step'svalueis not range-checked against the register's meaning. A wrong address is written to the wrong register. This is the single most dangerous property in the system, and it is an implementation gap rather than a safety net you can lean on. - Protection status. For clearance actions,
gate_reasonrefuses while anything else is still active on the device. - Execution confirmation.
verify: truegives read-back confirmation of a write. - Failure handling.
failure_actiondecides abort / skip / retry / recovery sequence.
Run it as a background job, not in the foreground of a session you might close:
python3 device_command_sequences.py --device <engine-key> --device-key <key> \
--task <action> --param power_kw=0 --json
Simulator testing must be the default. It is not merely good practice here — the REST path enforces it, and the physical path has no register validation to catch a mistake that the simulator would have caught for free.
Validating a Sequence Without Running It
curl -s -X POST "http://<ems-host>/api/v1/ems/command-sequences/validate" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"device_type": "<category>", "model": "<MODEL>", "data": { ... }}'
Because the register map is not consulted (configured_steps aside), validation confirms
structure and vocabulary. It does not confirm that an address is correct.
Troubleshooting
| Symptom | Cause |
|---|---|
<label> '<key>' has no endpoint configured |
Neither the device node nor the category's Modbus config has ip/port |
| Refused: not a simulator | The host string contains no simulation hint and authorize_physical was not passed |
Refused: target="physical" |
authorize_physical was not passed — the target alone is not enough |
A step reports ERROR rather than FAIL |
It timed out or errored, not a wrong value |
| A write lands on the wrong register | No register validation exists; check the address against the register map |
| Cancel seems to do nothing | A direct run_action cannot be cancelled; use start_test/stop_test |
| The test targets an address you did not expect | resolve_endpoint()["source"] names the config file the endpoint came from |
Implementation Gaps
- No register or range validation on steps. Addresses and values are not checked against the model's register map, on the simulator or on live plant.
- No cancellation on
run_action. Only the test path (start_test/stop_test) supports stopping a run in flight. - Simulator detection is by host name. A live device whose configured host happens to
contain
mock,fake,simulatoror similar would be treated as a simulation and would not requireauthorize_physical. Unlikely, but it is a string match rather than a declaration, so it is worth knowing the rule. - No PCS sequences exist, so the PCS sequences the brief describes cannot be tested — there is nothing to run.