Skip to content

device_duplicate.py — device folder manager

Reference for the script that keeps every device family's shared files in step with its per-unit folders, creates folders for new devices, and reports the state of the whole emsConfig device tree.

  • Script: EnergySyc_EMS_ES100Cx12S/device_duplicate.py
  • Runs against: emsConfig/ (default: the emsConfig folder beside the script)
  • Writes to: emsConfig/<family>/<prefix><n>/ and, on --scaffold, new folders
  • Never touches: per-device files, emsConfig/ems/ems_register_full.json, backup/ folders

1. The problem it solves

Every device family in emsConfig stores its files in the same two-part shape:

emsConfig/<family_dir>/
    <prefix>_main/          files shared by every unit of the family
    <prefix>1/              one folder per unit
    <prefix>2/
    ...
    <prefix>10/
    backup/                 retired files, never read

The shared half is the alarm codes and sequences/. The per-device half is the register map (and its model variants), the modbus configs and the lastValue template — those carry each unit's own identity and addresses.

That split creates three chores that are easy to get wrong by hand:

  1. When an alarm code changes, every unit folder needs the same new file.
  2. When a device is added to ems_control_setting.json, its folder has to be created and seeded before the build will accept it.
  3. When a family drifts — one unit folder holding a stale alarm code — nothing notices until the build or the runtime misbehaves.

device_duplicate.py does all three, and can report on them without writing anything.


2. Two names per family

A family has a directory and a folder prefix, and they are not always the same word. This is why FAMILIES is a mapping rather than a list:

--family directory under emsConfig/ folder prefix example folder
pcs pcs pcs pcs/pcs1/
bmu bmu bmu bmu/bmu1/
grid grid_energy_meter grid_meter grid_energy_meter/grid_meter1/
solar solar_energy_meter solar_meter solar_energy_meter/solar_meter1/
ffr ffr_energy_meter ffrMeter ffr_energy_meter/ffrMeter1/
ups ups ups ups/ups1/
door door_sensor door_sensor door_sensor/door_sensor1/
temperature temperature_sensor temperature_sensor temperature_sensor/temperature_sensor1/
flood flood_sensor flood_sensor flood_sensor/flood_sensor1/
smoke smoke_sensor smoke_sensor smoke_sensor/smoke_sensor1/
heat heat_sensor heat_sensor heat_sensor/heat_sensor1/
sensorB sensor_block sensorB sensor_block/sensorB1/
ac air_condition_unit ac air_condition_unit/ac1/

<prefix>_main/ is the shared master for that family. It is not counted as a device folder, even though it matches the <prefix>… naming.


3. When to use it

Situation Command
You changed an alarm code in <prefix>_main/ and want it everywhere --sync (the default)
You want to know whether anything has drifted, without changing files --check
You are about to sync and want to see the plan first --dry-run
You added a device to ems_control_setting.json --scaffold
You created an empty <prefix>11/ by hand and want it seeded --scaffold
Something looks wrong and you need the whole resolved picture --debug
You only care about one family add --family <name> to any of the above

Use it when

  • The shared half of a family changed (alarm codes, sequences/).
  • A device was added to or removed from the settings file.
  • You are diagnosing "which unit folder is out of step".
  • Before a build, as a cheap consistency gate — --check exits non-zero on drift.

Do not expect it to

  • Copy per-device files. Register maps, modbus configs and lastValue templates legitimately differ per unit; the script will not move them, and --scaffold deliberately creates a folder without them so you can supply the real ones.
  • Delete anything. There is no prune or remove action. Removing a device is a manual move into backup/.
  • Know a new family. Adding a whole new family means editing FAMILIES (and usually SETTINGS_GROUPS) in the script. --scaffold handles new devices inside a known family.
  • Touch the merged map. ems_register_full.json is produced by the two other scripts, not this one.

4. Command reference

python3 device_duplicate.py [action] [options]

Actions

Exactly one action is expected. If none is given, --sync runs.

Flag Writes? What it does
--sync yes Default. Copies every shared file from <prefix>_main/ into each <prefix><n>/ that differs or is missing it.
--check no Reports drift per family and per folder. Exits 1 if any drift was found, 0 if everything matches.
--dry-run no Prints what --sync would copy, without copying.
--scaffold yes Creates folders for devices declared in the settings file that have no folder, and fills device folders that exist but are empty. Prints the per-device files you must supply.
--debug no Dumps the resolved state of every family and exits 0.
--help no Full usage text.

--check and --dry-run overlap. If both are given, the script says so and behaves as --check.

Options

Flag Default Meaning
--config <dir> <script dir>/emsConfig The emsConfig directory to operate on.
--settings <rel> ems/ems_control_setting.json Settings file, relative to --config. Only used by --scaffold and --debug.
--family <name> all Restrict to one family. Accepts any key from the table in §2, plus all.

Exit codes

Code Meaning
0 Success. With --check: no drift.
1 --check only: drift was found.
2 The --config directory does not exist.

5. Where the device list comes from

--scaffold and --debug read ems_control_setting.json and compare its declared units against the folders on disk. Each settings group maps to a family through:

SETTINGS_GROUPS = { group: (unit-key stem, family dir, folder prefix) }

The stem is the word the settings file uses for a unit; the prefix is what the folder is called. They differ whenever names disagree:

Settings group unit key folder it maps to
pcs pcs1 pcs/pcs1/
bmu_units bmu1 bmu/bmu1/
grid_meters grid_meter1 grid_energy_meter/grid_meter1/
solar_meters solar_meter1 solar_energy_meter/solar_meter1/
ffr_meters ffrEnergyMeter1 ffr_energy_meter/ffrMeter1/
ups_units ups1 ups/ups1/
door_sensors door_sensor1 door_sensor/door_sensor1/
temperature_sensors temperature_sensor1 temperature_sensor/temperature_sensor1/
flood_sensors flood_sensor1 flood_sensor/flood_sensor1/
smoke_detectors smoke_detector1 smoke_sensor/smoke_sensor1/
heat_sensors heat_sensor1 heat_sensor/heat_sensor1/
sensor_blocks sensor_block1 sensor_block/sensorB1/
ac_units ac1 air_condition_unit/ac1/

Note the two rows where the settings key and the folder name genuinely disagree: smoke_detector1 lives in smoke_sensor1/, and sensor_block1 lives in sensorB1/. This is why the stem is tracked separately rather than derived.

Groups absent from SETTINGS_GROUPS are reported by --debug but never scaffolded, because their layout has not been confirmed. Currently reported as uncovered: bess_energy_meters, cooling_unit, frequency_meters, fss. Adding one is a deliberate act: confirm the directory and prefix, add the row, then --scaffold. Guessing would create wrongly-named folders.


6. Worked examples

Day-to-day: an alarm code changed

$ python3 device_duplicate.py --check
=== bmu : bmu/bmu_main -> 10 device folder(s) ===
      bmu_alarm_code.json
      bmu_alarm_code___bmu100.json
      ...
      already matching: 7 folder(s)
      out of step: 3 folder(s)
        bmu/bmu3    bmu_alarm_code.json differs
        bmu/bmu4    bmu_alarm_code.json differs
        bmu/bmu8    bmu_alarm_code.json missing
      (--check: nothing written)
...
DRIFT DETECTED - run without --check to refresh the folders
$ echo $?
1

Then apply it:

$ python3 device_duplicate.py

One family only

$ python3 device_duplicate.py --family bmu

A device was added to the settings file

Add bmu11 to general_settings.bmu_units, then:

$ python3 device_duplicate.py --scaffold --family bmu
scaffold: creating folders for devices declared in ems/ems_control_setting.json

=== bmu : bmu/ ===
  NEW    bmu11   (from bmu_units.bmu11  device_id=Cl1_P11)
        + bmu_alarm_code.json
        + bmu_alarm_code___bmu100.json
        + sequences/
         copied 4 shared file(s) from bmu_main/
         >>> NOW PUT THESE JSON FILES INSIDE bmu/bmu11/ <<<
             bmu_modbus_config.json
             bmu_register_full.json
             bmu_register_full___bmu100.json
             bmu_register_full___bmu200.json
             bmu_register_full___bmu300.json

The folder is created and seeded with the shared half. The files listed under the >>> NOW PUT THESE JSON FILES INSIDE … <<< banner are the per-device ones — the script cannot invent them, and the build will be incomplete until they exist.

An empty folder was created by hand

Creating bmu/bmu11/ yourself and running --scaffold gives:

=== bmu : bmu/ ===
  EMPTY  bmu/bmu11   (exists but holds nothing)
        + bmu_alarm_code.json
        ...
         copied 4 shared file(s) from bmu_main/
         >>> NOW PUT THESE JSON FILES INSIDE bmu/bmu11/ <<<
             ...

An empty folder is one holding no files anywhere underneath — a folder containing only an empty sequences/ still counts as empty. Running --scaffold twice is harmless: the second run reports that every declared device already has a folder.

Diagnosing the whole tree

$ python3 device_duplicate.py --debug
config: /…/emsConfig

resolved config: /…/emsConfig
settings file:   /…/emsConfig/ems/ems_control_setting.json

families: 13
---- ups
     dir            ups/ups
     shared files   4  ups_alarm_code.json, ups_alarm_code___ups_10k.json, …
     shared dirs    sequences
     device folders 1  ups1
     declared       1  ups1
     per-device     ups_modbus_config.json, ups_register_full.json, …
...
settings groups with no scaffold mapping (4): bess_energy_meters, cooling_unit, frequency_meters, fss

--debug is the fastest way to answer "what does the script think the layout is" without changing anything.


7. Reading the output

Label Meaning
already matching: N folder(s) Units whose shared files are byte-identical to <prefix>_main/.
out of step: N folder(s) Units with a shared file missing or differing.
EMPTY <path> A device folder with no files anywhere underneath; --scaffold seeds it.
SKIP: <path>_main does not exist The family has no shared master, so there is nothing to sync. If this is unexpected, the family's folder was renamed or never created.
SKIP: <prefix>_main/ has no shared files The master exists but is empty.
MISSING (in --debug) Declared in the settings file with no folder on disk. Run --scaffold.
not declared (in --debug) A folder on disk with no corresponding settings entry. The build will not see it.
>>> NOW PUT THESE JSON FILES INSIDE … <<< Per-device files you must supply by hand.

Comparison is by SHA-256, so a whitespace-only change counts as drift. Files matching the history pattern — .bak, .back, .backup, backupNN, _backup, _clean_, write_queue, .orig, ~ — are ignored entirely: never synced, never reported.


8. Relationship to the other scripts

Three scripts share this tree, and they do different jobs:

Script Direction Writes
generate_ems_register_full.py device folders → device maps ems/ems_register_full.json
ems_merge_register_map.py device maps + settings → EMS map ems/ems_register_full.json, and annotates ems/ems_control_setting.json
device_duplicate.py <prefix>_main/ → <prefix><n>/ device folders only

device_duplicate.py is the one that never touches the settings file or the merged map. Run it before the other two when the shared files changed, so the build sees a consistent tree.

Two warnings carried over from the build scripts:

  • ems_merge_register_map.py regenerates the whole register_map block of every settings unit it can map, including ems_base_address / ems_end_address / register_count. Those are EMS-side allocations in the 30xxx space, computed by that script — not device-side spans. Do not hand-write them.
  • A settings group that maps through GROUP_TO_BLOCK but has no matching entry in that script's BLOCKS table is silently skipped during annotation. The symptom is a register_map that keeps whatever was last written by hand.

9. Conventions for this script

Every Python script in this project must provide both --help and --debug.

  • --help — not the bare argparse default: a real docstring describing the layout and behaviour, with an epilog of usage examples.
  • --debug — dumps resolved state (paths, whether settings loaded, per-family counts, declared vs on-disk, missing and empty items) and exits without writing.

All three scripts implement both — device_duplicate.py, generate_ems_register_full.py and ems_merge_register_map.py. Run <script> --debug before a build to see the resolved state without writing.

Two more invariants worth preserving:

  • Shared files are derived from <prefix>_main/, never hardcoded. Adding a file to a master teaches the script about it automatically.
  • The per-device list shown by --scaffold is read off sibling folders, not hardcoded, so a family that grows a new per-device file type is reported correctly with no code change.

10. Troubleshooting

A family reports SKIP: … does not exist. The directory or its <prefix>_main/ is missing or misnamed. Check the family row in §2 against what is on disk; a renamed directory needs a FAMILIES edit.

Everything is "out of step" after a legitimate per-device change. Per-device files are not compared, so this means a shared file differs in those folders too. Check whether you edited a file inside a device folder that should have been edited in <prefix>_main/.

--scaffold says a device is NEW but the folder exists. The folder name does not match the expected <prefix><n>. --scaffold matches strictly on the prefix plus digits; a folder named bmu_11 will not be seen as bmu11.

--scaffold refuses to act. It prints cannot scaffold <name> -- <prefix>_main/ does not exist, so the layout is unknown. Create the master folder first, then re-run.

A new folder was created but the build ignores it. The build's own tables are separate. A brand-new device needs its block declared in generate_ems_register_full.py (and usually ems_merge_register_map.py) as well as a folder.

Unsure what the script sees? --debug first, always. It resolves and reports without writing.