Skip to content

Frequency Meter Client — frequencyMeter_client_save_data_v2.py

frequencyMeter_client_save_data_v2.py polls the frequency meter over Modbus TCP and persists its samples to both databases. It replaces the fixed-rate v1 loop.

The meter is the highest-rate device in the stack: it is designed to sample at 10 ms (100 Hz), roughly two orders of magnitude faster than the other meters. That constraint drives every design decision in this client.

Why this client is different

Concern Other meter clients (grid, solar) Frequency meter client
Sample rate seconds 10–100 ms (configurable)
Persistence one row per sample one row per batch of samples
Delivery HTTP POST to insert-multi-device-data direct multi-row INSERT into two databases
Rate source fixed in code sampling_interval_ms from config
Register reads one round trip per register group coalesced contiguous blocks

At 100 Hz a per-sample round trip to the database, or a per-sample HTTP POST, cannot keep up. Buffering and bulk inserts are what make 10 ms sustainable.

Configuration

All timing and connection settings come from emsConfig/ems/ems_control_setting.json, under general_settings.frequency_meters.frequency_meter1:

{
  "frequency_meter1": {
    "communication": { "protocol": "modbus-tcp", "ip_address": "" },
    "device": {
      "ip": "bess-simulator-backend",
      "port": 1515,
      "unit_id": 5,
      "log_level": "INFO",
      "endianness": "big",
      "ems_key": "ems",
      "scale": 1,
      "datatype": "BITMAP32",
      "sampling_interval_ms": 10,
      "batch_size": 1800,
      "batch_interval_ms": 180000,
      "site_id": 1
    },
    "status": "active",
    "device_id": "FREQUENCY-METER-001",
    "model": "ME231"
  }
}

Timing keys

Key Meaning Range enforced Default
poll_interval_ms (alias sampling_interval_ms) Period between Modbus reads 10 – 3000 10
save_interval_ms (alias batch_interval_ms) Buffer window: one flush covers this much time max(1000, poll) – 86400000 180000 (3 min)
batch_size derived, not read: save_interval_ms / poll_interval_ms, capped at 1800 1 – 1800 derived

How the two keys combine. The Writer flushes on a fixed save_interval_ms cadence, then slices that window into rows of batch_size samples each. At 10 ms sampling over a 180 s window the window holds 18 000 samples, batch_size derives to 1800, and each flush produces 10 rows of 18 seconds each. batch_size does not control when data is written; it controls how one written window is divided.

The device block accepts both spellings. poll_interval_ms (sampling period) and save_interval_ms (buffer window) take precedence, with sampling_interval_ms and batch_interval_ms accepted as aliases; anything missing falls back to frequencyMeter_modbus_config.json and then to the defaults.

Device identity

gateway_id and device_id are not taken from ems_key. They come from the shared resolver ems_device_identity.require_device_identity("frequency_meter", key), which returns:

Field Resolved value Source
gateway_id GTW-ems100CX metadata.gateway_id in ems_control_setting.json
device_id FREQUENCY-METER-001 the device entry / frequencyMeter_modbus_config.json
model ME231 the device entry

See EMS Device ID, Model and Gateway Resolution for the full precedence rules.

ems_key is not a gateway

ems_key is the EMS settings key. Using it as gateway_id writes rows that the frontend can never retrieve, because the chart queries filter on the active gateway (GTW-ems100CX), not on ems. This was the original bug in this client and is why identity now goes through the shared resolver.

Switching the meter off

status (active / inactive) is honoured while the client runs, so the meter can be switched off from the frontend without touching the container. The process itself always stays up, ready for the meter to come back:

status Reader Writer
active samples every poll_interval_ms flushes every save_interval_ms
anything else waits, with no connection and no Modbus traffic writes the buffered tail once, then stays quiet

The change is noticed within a second and logged once per stretch, not as a heartbeat:

[IDLE] frequency meter is now 'inactive'; polling and saving stopped
[IDLE] frequency meter is 'inactive'; polling paused
[INFO] frequency meter is now active; polling and saving resume

This is why the flag is re-read rather than read once at start-up: a running client used to keep polling and keep inserting rows after the meter had been set to inactive.

Register reads

The register map is model-resolved: resolve_model_file() maps frequencyMeter_register.json to a variant such as frequencyMeter_register___me231.json, driven by emsConfig/selected_model.json.

The client does not read registers one at a time. It builds coalesced read blocks from the map:

  • MODBUS_MAX_SPAN = 125 — the protocol limit for a single read.
  • MODBUS_GROUP_GAP = 64 — adjacent registers closer than this are merged into one read.

For the current ME231 map this collapses 12 mapped registers into 5 reads of 140 registers total, instead of 12 separate round trips per sample.

Read list

Not every mapped register is worth a round trip at 10 ms, so emsConfig/frequency_meter/frequencyMeter_read_list.json decides which registers are polled:

{
  "poll_registers": [
    "frequency.frequency_hz",
    "power.active_power_total_kw",
    "power.reactive_power_total_kvar",
    "voltages.voltage_ab_v",
    "currents.current_a_a",
    "power_factor.pf_total",
    "alarms.alarms",
    "alarms.digital_input_status_validity_base"
  ]
}
  • A name may be written as category.register, as the bare register name (frequency_hz, alarms), or as the flat key (frequencyMeter_frequency_hz).
  • The file is model-resolved like the map, so frequencyMeter_read_list___me231.json would win over the base file. One file already covers every shipped model, because ME231, P8000 and PM5560 share these register names and differ only in addresses.
  • A missing file, an empty list, or ["*"] means "poll every register in the map".
  • A name that is not in the map is reported and ignored; if nothing matches, the client falls back to polling every register rather than writing empty samples.

The shipped list leaves out the four cumulative energy counters: at 10 ms they add nothing, and their addresses (2706, and 15000–15020) sit far from the rest, so each group costs a separate read on every single sample.

Read list Registers Coalesced reads per sample
absent (poll everything) 12 5
shipped default 8 3

Startup names the file and the counts it applied:

[INFO] frequencyMeter_read_list.json: polling 8 of 12 register(s)
[INFO] model map resolved; 8 registers in 3 coalesced read(s), endianness=big

Execution model

Two threads, connected by a bounded FIFO:

flowchart TD
    A["Reader thread<br/>(freq-reader)"] -->|decode_sample| B["SampleBuffer<br/>deque, capacity 400000"]
    B -->|drain| C["Writer thread<br/>(freq-writer)"]
    C -->|chunks of batch_size| D["DbSink"]
    D --> E[("ems_backend_db<br/>frequency_meter_measurements")]
    D --> F[("ems_db<br/>frequency_meter_readings")]
  • Reader — sleeps for sampling_interval_ms, performs the coalesced reads, decodes each register into a sample dict, and appends to the buffer. It never blocks on the database.
  • SampleBuffer — a bounded deque. On overflow the oldest item is dropped and counted, so overflow is visible rather than silent.
  • Writer — wakes every batch_interval_ms, drains the buffer, slices it into batch_size chunks, and writes each chunk to both databases.
  • dropped=0 in the flush log line is the health signal: the reader was never starved.

Sample shape

Each sample is a flat dict keyed by the register-map names, all under the frequencyMeter_ prefix:

{
  "frequencyMeter_frequency_hz": 50.0,
  "frequencyMeter_voltage_ab_v": 0.0,
  "frequencyMeter_voltage_bc_v": 0.0,
  "frequencyMeter_voltage_ca_v": 0.0,
  "frequencyMeter_current_a_a": 0.0,
  "frequencyMeter_active_power_total_kw": 150.0,
  "frequencyMeter_reactive_power_total_kvar": 0.0,
  "frequencyMeter_pf_total": 1.0,
  "frequencyMeter_active_energy_delivered_kwh": 0.0,
  "frequencyMeter_active_energy_received_kwh": 0.0,
  "frequencyMeter_active_energy_delivered_plus_received_kwh": 0.0,
  "frequencyMeter_active_energy_delivered_minus_received_kwh": 0.0,
  "frequencyMeter_digital_input_status_validity_base": 0,
  "frequencyMeter_alarms": 0,
  "timestamp": 1790420751946
}

A sample carries exactly the registers in the read list, plus timestamp, so this shape is the full ME231 map rather than the polled subset.

timestamp is epoch milliseconds, matching the x-axis convention the frontend chart expects.

Databases

The client writes to two tables in one flush.

Backend — ems_backend_db.frequency_meter_measurements

Created on demand if absent.

Column Type Contents
time bigint / primary key first sample's timestamp (ms), per chunk
metadata json / text JSON blob of the batch envelope

Frontend — ems_db.frequency_meter_readings

The table is created by the Laravel migration 2026_09_26_000001_create_frequency_meter_readings_table.php and is owned by the ems-frontend application (App\Models\FrequencyMeterReading).

Column Contents
device_id FREQUENCY-METER-001
gateway_id GTW-ems100CX
component_type frequency_meters
readings JSON batch envelope (see below)
site_id from device.site_id
text_01, text_02 reserved, empty

Batch envelope

Both tables store the same envelope per chunk:

{
  "batch_start_ms": 1790420751946,
  "batch_end_ms": 1790420753496,
  "sampling_interval_ms": 10,
  "batch_size": 1800,
  "sample_count": 155,
  "samples": [ { "timestamp": 1790420751946, "frequencyMeter_frequency_hz": 50.0, ... } ]
}

samples is capped at batch_size entries because the Writer slices before writing. The frequency_meters chart config on the frontend expects frequencyMeter_frequency_hz and frequencyMeter_active_power_total_kw, and reads them out of this envelope — see Frequency Meter Data Pipeline.

Running

The client is launched by start_ems_client_in_docker.sh:

start_py "frequency_meter_client" frequencyMeter_client_save_data_v2.py

Logging and health

Log file: logs/frequency_meter_client.log. Startup emits one line per resolved setting, and each flush emits a summary:

[10:09:14.760] [INFO] model map resolved; 12 registers in 5 coalesced read(s), endianness=big
[10:09:14.760] [INFO] sampling=10 ms | batch_size=1800 | batch_interval=180000 ms | target=bess-simulator-backend:1515
[10:09:14.794] [INFO] DB backend detected: mysql (backend db=ems_backend_db)
[10:09:14.929] [OK] ensured ems_backend_db.frequency_meter_measurements
[10:09:14.931] [OK] ems_db.frequency_meter_readings present
[10:09:14.933] [OK] connected to frequency meter at bess-simulator-backend:1515 (unit 5)
[10:12:19.455] [FLUSH #1] samples=17979 backend_rows=10 frontend_rows=10 dropped=0 db_ms=4126.2

Reading the flush line:

  • samples — samples captured in the window. Over a 180 s window at 10 ms the ideal is 18 000; 17 979 is 99.88 %.
  • backend_rows / frontend_rows — rows written to each database; both should be equal to ceil(samples / batch_size).
  • dropped — samples discarded due to a full buffer. Should be 0. A non-zero value means the Writer cannot keep up: raise batch_interval_ms down, raise sampling_interval_ms, or increase batch_size.
  • db_ms — milliseconds spent writing the flush.

Changing the sample rate, or switching the meter off

Neither one needs a restart. The client re-reads ems_control_setting.json at least once a second and applies the change on the next cycle:

  1. Set device.poll_interval_ms to the new value (10–3000 ms).
  2. Optionally set device.save_interval_ms; save_interval_ms / poll_interval_ms is what becomes the row size.

The reload is logged with the fields that changed:

[INFO] settings reloaded: poll_interval_ms=100, save_interval_ms=180000
[INFO] frequency meter is now active; polling and saving resume

Values that change what a sample means — datatype, endianness, model, device_id, gateway_id — are reported and then ignored, because applying them mid-window would mix two meanings into one row. Those still need the restart.

If the interval changes, revisit save_interval_ms, otherwise the points drawn per chart will drift.

Changing the model

The client resolves the model once at startup, so a model change restarts the client — the same step as above:

  1. Set the model for frequencyMeter_register.json in emsConfig/selected_model.json (either the global model or a by_register_file override).
  2. Restart the client by restarting ems-backend.
  3. Apply the matching change on the simulator side and restart it, so both sides serve and read the same register layout. The simulator has the same startup-resolution constraint; see the simulator repository's docs/model_selection.md for its selection file and restart step.

Because the model fixes the register addresses, a mismatch does not degrade gracefully: the client reads the wrong addresses or the server answers IllegalAddress, and the chart stays empty.

Verification

# backend rows
docker exec mariadb mysql -u scada_user -p -N -B -e \
  "SELECT COUNT(*) FROM ems_backend_db.frequency_meter_measurements;"

# frontend rows, keyed correctly
docker exec mariadb mysql -u scada_user -p -N -B -e \
  "SELECT component_type, gateway_id, device_id, COUNT(*)
     FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;"

Expected identity in the second query: frequency_meters | GTW-ems100CX | FREQUENCY-METER-001. Any other gateway_id means identity resolution is misconfigured.