Skip to content

How to Test Device Command Sequences

Verify a command sequence against the BESS simulator before it ever touches real equipment, and understand exactly what the code does and does not let you do on live plant.


Purpose

Prove that a sequence does what its author intended: the right registers, the right encodings, the right values, and the right outcome when a step fails. The test harness is built for this and enforces simulator-first operation in code.


Prerequisites

  • The EMS backend running, with pyModbusTCP available.
  • A device whose model defines the sequence under test — confirm with --device <engine-key> --list-actions.
  • For simulator testing: a Modbus endpoint whose host is recognisably a simulation.
  • For physical testing: the two gates below, satisfied deliberately.

Relevant Files

Path Role
command_sequence_test.py The harness: run_test, start_test, stop_test
device_command_sequences.py The engine and its CLI
ems_settings_api.py The REST test endpoints
protocol_adapters.py The adapters the test drives
resources/views/ The command-sequence editor and execution screens

A. Simulator Testing

What counts as a simulator

The harness decides from the host name, and it is deliberately generous about what looks like a simulation:

_SIMULATOR_HINTS = ("simulator", "simulated", "sim-backend", "mock", "fake")

def is_simulator_host(host) -> bool:
    text = str(host or "").strip().lower()
    if not text:
        return False
    if text in _LOOPBACK:
        return True
    return any(hint in text for hint in _SIMULATOR_HINTS)

Loopback addresses always count. Otherwise the host string must contain one of those hints. The resolved endpoint reports this as "simulator": true|false, and the endpoint is read from the same configuration node the EMS itself polls — "no second, drifting notion of the device address." So a test targets exactly what the EMS is driving, not a separate address someone typed in.

Steps

  1. Start the required services. The backend must be up, and the simulator must be listening on the device's configured host and port.
  2. Identify the device's endpoint — where it is and whether the harness considers it a simulation:
python3 -c "from command_sequence_test import resolve_endpoint; import json; \
  print(json.dumps(resolve_endpoint('<device_type>', '<device_key>'), indent=2))"

Check simulator and source in the output. source names the exact config path the endpoint came from, which is how you prove you are testing what you think.

  1. Confirm the model and its sequences:
python3 device_command_sequences.py --device <engine-key> --list-models
python3 device_command_sequences.py --device <engine-key> --model <MODEL> --list-actions
  1. Look at the steps before running them. configured_steps(device_type, model, action) returns the sequence's steps as the engine will execute them, so a wrong address is visible before a single write:
python3 -c "from command_sequence_test import configured_steps; import json; \
  print(json.dumps(configured_steps('<category>','<MODEL>','<action>'), indent=2))"
  1. Open the command sequence editor and select the target device.
  2. Select the sequence — the picker lists the model's real actions keys.
  3. Enter the dynamic input parameters. Each ${name} placeholder in the steps must be supplied; an unsupplied placeholder cannot resolve.
  4. Validate without executing — this writes nothing: POST /api/v1/ems/command-sequences/validate with device_type, model and data.
  5. Execute against the simulator.
  6. Monitor each step. Steps report one of:

    _STEP_RESULT = {"ok": "PASS", "command": "PASS", "skipped": "SKIPPED",
                    "failed": "FAIL", "timeout": "ERROR", "error": "ERROR"}
    
  7. Verify register writes and responses. A step with verify: true reads the register back; the measured value is reported alongside the expected one.

  8. Check delays and conditions took effect in order.
  9. Review the logs, and the --debug output if a step is unexpected.
  10. Read the verdict. The report's status is PASS or FAIL, with elapsed time stamped. FAIL when any row is FAIL or ERROR.

The default path is the safe one

Nothing above needs a flag. The harness refuses a non-simulator host unless the caller explicitly passes authorize_physical=True, so the default path is the simulator path.


B. Testing from Python

from command_sequence_test import resolve_endpoint, run_test, start_test, stop_test

# 1. Show where this will run, and whether it is a simulation.
endpoint = resolve_endpoint("ev_charger", "EV-1")
print(endpoint["host"], endpoint["port"], "simulator:", endpoint["simulator"])

# 2. Synchronous run against the simulator.
report = run_test("ev_charger", "set_current_limit",
                  device_key="EV-1",
                  parameters={"current_a": 16})
print("status:", report["status"])              # PASS / FAIL / SKIPPED / STOPPED
for row in report["steps"]:                     # the per-step rows live under "steps"
    print(" ", row["status"], row["step"], row["actual"])

# 3. Asynchronous run, for a sequence long enough to want to watch it.
job = start_test("ev_charger", "set_current_limit",
                 device_key="EV-1", parameters={"current_a": 16})
print("job id:", job["test_id"])                # the job key is "test_id", not "id"

# 4. Cancel it.
print(stop_test(job["test_id"]))

start_test is the asynchronous entry point and returns a job record whose test_id is what stop_test takes. That pairing is the supported way to cancel — there is no cancellation on run_action, so a long sequence you may need to abort should go through the test path. A cancelled run comes back with status: "STOPPED" and the steps that never ran marked SKIPPED.

Errors you will see

Exception Meaning
SequenceTestError Base class for the three below — the test could not be set up, often no endpoint configured
TestNotFound No such test id
TestConflict A conflicting test is already running on that device
SequenceStopped The test was cancelled. Imported from pcs_power_sequence, not from command_sequence_test

A device with no usable endpoint raises <label> '<device_key>' has no endpoint configured.


C. Live Device Testing

Physical execution is gated by two independent checks, both of which must be satisfied:

if target == "physical" and not authorize_physical:
    # refused

if not endpoint["simulator"] and not authorize_physical:
    # refused: "...because it is not a simulator. Pass authorize_physical to ..."

The module's own reasoning: "asking for physical execution is never enough on its own." Declaring target="physical" does not authorise anything — the flag is required separately, and a host that merely fails to look like a simulator still needs it. A typo in a host name cannot accidentally reach live equipment.

Before a physical run, confirm each of these:

  • User authorization. authorize_physical=True passed deliberately by a person who intends it.
  • EMS operational mode. The mode gates clearance actions through gate_reason's allowed_modes; a mode that forbids clearance actions will refuse them.
  • Device communication. The endpoint resolves and the TCP connection opens; otherwise cannot open Modbus connection to host:port.
  • Safety interlocks. Whatever the site relies on; the harness does not substitute for them.
  • Register and parameter validation. ⚠️ There is none. A step's address is not cross-checked against the model's register map, and a step's value is not range-checked against the register's meaning. A wrong address is written to the wrong register. This is the single most dangerous property in the system, and it is an implementation gap rather than a safety net you can lean on.
  • Protection status. For clearance actions, gate_reason refuses while anything else is still active on the device.
  • Execution confirmation. verify: true gives read-back confirmation of a write.
  • Failure handling. failure_action decides abort / skip / retry / recovery sequence.

Run it as a background job, not in the foreground of a session you might close:

python3 device_command_sequences.py --device <engine-key> --device-key <key> \
    --task <action> --param power_kw=0 --json

Simulator testing must be the default. It is not merely good practice here — the REST path enforces it, and the physical path has no register validation to catch a mistake that the simulator would have caught for free.


Validating a Sequence Without Running It

curl -s -X POST "http://<ems-host>/api/v1/ems/command-sequences/validate" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"device_type": "<category>", "model": "<MODEL>", "data": { ... }}'

Because the register map is not consulted (configured_steps aside), validation confirms structure and vocabulary. It does not confirm that an address is correct.


Troubleshooting

Symptom Cause
<label> '<key>' has no endpoint configured Neither the device node nor the category's Modbus config has ip/port
Refused: not a simulator The host string contains no simulation hint and authorize_physical was not passed
Refused: target="physical" authorize_physical was not passed — the target alone is not enough
A step reports ERROR rather than FAIL It timed out or errored, not a wrong value
A write lands on the wrong register No register validation exists; check the address against the register map
Cancel seems to do nothing A direct run_action cannot be cancelled; use start_test/stop_test
The test targets an address you did not expect resolve_endpoint()["source"] names the config file the endpoint came from

Implementation Gaps

  1. No register or range validation on steps. Addresses and values are not checked against the model's register map, on the simulator or on live plant.
  2. No cancellation on run_action. Only the test path (start_test/stop_test) supports stopping a run in flight.
  3. Simulator detection is by host name. A live device whose configured host happens to contain mock, fake, simulator or similar would be treated as a simulation and would not require authorize_physical. Unlikely, but it is a string match rather than a declaration, so it is worth knowing the rule.
  4. No PCS sequences exist, so the PCS sequences the brief describes cannot be tested — there is nothing to run.