Frequency Meter Client — frequencyMeter_client_save_data_v2.py
frequencyMeter_client_save_data_v2.py polls the frequency meter over Modbus TCP
and persists its samples to both databases. It replaces the fixed-rate v1
loop.
The meter is the highest-rate device in the stack: it is designed to sample at 10 ms (100 Hz), roughly two orders of magnitude faster than the other meters. That constraint drives every design decision in this client.
Why this client is different
| Concern | Other meter clients (grid, solar) | Frequency meter client |
|---|---|---|
| Sample rate | seconds | 10–100 ms (configurable) |
| Persistence | one row per sample | one row per batch of samples |
| Delivery | HTTP POST to insert-multi-device-data |
direct multi-row INSERT into two databases |
| Rate source | fixed in code | sampling_interval_ms from config |
| Register reads | one round trip per register group | coalesced contiguous blocks |
At 100 Hz a per-sample round trip to the database, or a per-sample HTTP POST, cannot keep up. Buffering and bulk inserts are what make 10 ms sustainable.
Configuration
All timing and connection settings come from
emsConfig/ems/ems_control_setting.json, under
general_settings.frequency_meters.frequency_meter1:
{
"frequency_meter1": {
"communication": { "protocol": "modbus-tcp", "ip_address": "" },
"device": {
"ip": "bess-simulator-backend",
"port": 1515,
"unit_id": 5,
"log_level": "INFO",
"endianness": "big",
"ems_key": "ems",
"scale": 1,
"datatype": "BITMAP32",
"sampling_interval_ms": 10,
"batch_size": 1800,
"batch_interval_ms": 180000,
"site_id": 1
},
"status": "active",
"device_id": "FREQUENCY-METER-001",
"model": "ME231"
}
}
Timing keys
| Key | Meaning | Range enforced | Default |
|---|---|---|---|
poll_interval_ms (alias sampling_interval_ms) |
Period between Modbus reads | 10 – 3000 | 10 |
save_interval_ms (alias batch_interval_ms) |
Buffer window: one flush covers this much time | max(1000, poll) – 86400000 | 180000 (3 min) |
batch_size |
derived, not read: save_interval_ms / poll_interval_ms, capped at 1800 |
1 – 1800 | derived |
How the two keys combine. The Writer flushes on a fixed
save_interval_mscadence, then slices that window into rows ofbatch_sizesamples each. At 10 ms sampling over a 180 s window the window holds 18 000 samples,batch_sizederives to 1800, and each flush produces 10 rows of 18 seconds each.batch_sizedoes not control when data is written; it controls how one written window is divided.
The device block accepts both spellings. poll_interval_ms (sampling period) and
save_interval_ms (buffer window) take precedence, with sampling_interval_ms
and batch_interval_ms accepted as aliases; anything missing falls back to
frequencyMeter_modbus_config.json and then to the defaults.
Device identity
gateway_id and device_id are not taken from ems_key. They come from the
shared resolver ems_device_identity.require_device_identity("frequency_meter", key),
which returns:
| Field | Resolved value | Source |
|---|---|---|
gateway_id |
GTW-ems100CX |
metadata.gateway_id in ems_control_setting.json |
device_id |
FREQUENCY-METER-001 |
the device entry / frequencyMeter_modbus_config.json |
model |
ME231 |
the device entry |
See EMS Device ID, Model and Gateway Resolution for the full precedence rules.
ems_key is not a gateway
ems_key is the EMS settings key. Using it as gateway_id writes rows
that the frontend can never retrieve, because the chart queries filter on
the active gateway (GTW-ems100CX), not on ems. This was the original
bug in this client and is why identity now goes through the shared resolver.
Switching the meter off
status (active / inactive) is honoured while the client runs, so the meter
can be switched off from the frontend without touching the container. The process
itself always stays up, ready for the meter to come back:
status |
Reader | Writer |
|---|---|---|
active |
samples every poll_interval_ms |
flushes every save_interval_ms |
| anything else | waits, with no connection and no Modbus traffic | writes the buffered tail once, then stays quiet |
The change is noticed within a second and logged once per stretch, not as a heartbeat:
[IDLE] frequency meter is now 'inactive'; polling and saving stopped
[IDLE] frequency meter is 'inactive'; polling paused
[INFO] frequency meter is now active; polling and saving resume
This is why the flag is re-read rather than read once at start-up: a running
client used to keep polling and keep inserting rows after the meter had been set
to inactive.
Register reads
The register map is model-resolved: resolve_model_file() maps
frequencyMeter_register.json to a variant such as
frequencyMeter_register___me231.json, driven by emsConfig/selected_model.json.
The client does not read registers one at a time. It builds coalesced read blocks from the map:
MODBUS_MAX_SPAN = 125— the protocol limit for a single read.MODBUS_GROUP_GAP = 64— adjacent registers closer than this are merged into one read.
For the current ME231 map this collapses 12 mapped registers into 5 reads of 140 registers total, instead of 12 separate round trips per sample.
Read list
Not every mapped register is worth a round trip at 10 ms, so
emsConfig/frequency_meter/frequencyMeter_read_list.json decides which registers
are polled:
{
"poll_registers": [
"frequency.frequency_hz",
"power.active_power_total_kw",
"power.reactive_power_total_kvar",
"voltages.voltage_ab_v",
"currents.current_a_a",
"power_factor.pf_total",
"alarms.alarms",
"alarms.digital_input_status_validity_base"
]
}
- A name may be written as
category.register, as the bare register name (frequency_hz,alarms), or as the flat key (frequencyMeter_frequency_hz). - The file is model-resolved like the map, so
frequencyMeter_read_list___me231.jsonwould win over the base file. One file already covers every shipped model, because ME231, P8000 and PM5560 share these register names and differ only in addresses. - A missing file, an empty list, or
["*"]means "poll every register in the map". - A name that is not in the map is reported and ignored; if nothing matches, the client falls back to polling every register rather than writing empty samples.
The shipped list leaves out the four cumulative energy counters: at 10 ms they add nothing, and their addresses (2706, and 15000–15020) sit far from the rest, so each group costs a separate read on every single sample.
| Read list | Registers | Coalesced reads per sample |
|---|---|---|
| absent (poll everything) | 12 | 5 |
| shipped default | 8 | 3 |
Startup names the file and the counts it applied:
[INFO] frequencyMeter_read_list.json: polling 8 of 12 register(s)
[INFO] model map resolved; 8 registers in 3 coalesced read(s), endianness=big
Execution model
Two threads, connected by a bounded FIFO:
flowchart TD
A["Reader thread<br/>(freq-reader)"] -->|decode_sample| B["SampleBuffer<br/>deque, capacity 400000"]
B -->|drain| C["Writer thread<br/>(freq-writer)"]
C -->|chunks of batch_size| D["DbSink"]
D --> E[("ems_backend_db<br/>frequency_meter_measurements")]
D --> F[("ems_db<br/>frequency_meter_readings")]
- Reader — sleeps for
sampling_interval_ms, performs the coalesced reads, decodes each register into a sample dict, and appends to the buffer. It never blocks on the database. - SampleBuffer — a bounded
deque. On overflow the oldest item is dropped and counted, so overflow is visible rather than silent. - Writer — wakes every
batch_interval_ms, drains the buffer, slices it intobatch_sizechunks, and writes each chunk to both databases. dropped=0in the flush log line is the health signal: the reader was never starved.
Sample shape
Each sample is a flat dict keyed by the register-map names, all under the
frequencyMeter_ prefix:
{
"frequencyMeter_frequency_hz": 50.0,
"frequencyMeter_voltage_ab_v": 0.0,
"frequencyMeter_voltage_bc_v": 0.0,
"frequencyMeter_voltage_ca_v": 0.0,
"frequencyMeter_current_a_a": 0.0,
"frequencyMeter_active_power_total_kw": 150.0,
"frequencyMeter_reactive_power_total_kvar": 0.0,
"frequencyMeter_pf_total": 1.0,
"frequencyMeter_active_energy_delivered_kwh": 0.0,
"frequencyMeter_active_energy_received_kwh": 0.0,
"frequencyMeter_active_energy_delivered_plus_received_kwh": 0.0,
"frequencyMeter_active_energy_delivered_minus_received_kwh": 0.0,
"frequencyMeter_digital_input_status_validity_base": 0,
"frequencyMeter_alarms": 0,
"timestamp": 1790420751946
}
A sample carries exactly the registers in the read list, plus timestamp, so
this shape is the full ME231 map rather than the polled subset.
timestamp is epoch milliseconds, matching the x-axis convention the
frontend chart expects.
Databases
The client writes to two tables in one flush.
Backend — ems_backend_db.frequency_meter_measurements
Created on demand if absent.
| Column | Type | Contents |
|---|---|---|
time |
bigint / primary key | first sample's timestamp (ms), per chunk |
metadata |
json / text | JSON blob of the batch envelope |
Frontend — ems_db.frequency_meter_readings
The table is created by the Laravel migration
2026_09_26_000001_create_frequency_meter_readings_table.php and is owned by the
ems-frontend application (App\Models\FrequencyMeterReading).
| Column | Contents |
|---|---|
device_id |
FREQUENCY-METER-001 |
gateway_id |
GTW-ems100CX |
component_type |
frequency_meters |
readings |
JSON batch envelope (see below) |
site_id |
from device.site_id |
text_01, text_02 |
reserved, empty |
Batch envelope
Both tables store the same envelope per chunk:
{
"batch_start_ms": 1790420751946,
"batch_end_ms": 1790420753496,
"sampling_interval_ms": 10,
"batch_size": 1800,
"sample_count": 155,
"samples": [ { "timestamp": 1790420751946, "frequencyMeter_frequency_hz": 50.0, ... } ]
}
samples is capped at batch_size entries because the Writer slices before
writing. The frequency_meters chart config on the frontend expects
frequencyMeter_frequency_hz and frequencyMeter_active_power_total_kw, and
reads them out of this envelope — see
Frequency Meter Data Pipeline.
Running
The client is launched by start_ems_client_in_docker.sh:
start_py "frequency_meter_client" frequencyMeter_client_save_data_v2.py
Logging and health
Log file: logs/frequency_meter_client.log. Startup emits one line per resolved
setting, and each flush emits a summary:
[10:09:14.760] [INFO] model map resolved; 12 registers in 5 coalesced read(s), endianness=big
[10:09:14.760] [INFO] sampling=10 ms | batch_size=1800 | batch_interval=180000 ms | target=bess-simulator-backend:1515
[10:09:14.794] [INFO] DB backend detected: mysql (backend db=ems_backend_db)
[10:09:14.929] [OK] ensured ems_backend_db.frequency_meter_measurements
[10:09:14.931] [OK] ems_db.frequency_meter_readings present
[10:09:14.933] [OK] connected to frequency meter at bess-simulator-backend:1515 (unit 5)
[10:12:19.455] [FLUSH #1] samples=17979 backend_rows=10 frontend_rows=10 dropped=0 db_ms=4126.2
Reading the flush line:
samples— samples captured in the window. Over a 180 s window at 10 ms the ideal is 18 000; 17 979 is 99.88 %.backend_rows/frontend_rows— rows written to each database; both should be equal toceil(samples / batch_size).dropped— samples discarded due to a full buffer. Should be 0. A non-zero value means the Writer cannot keep up: raisebatch_interval_msdown, raisesampling_interval_ms, or increasebatch_size.db_ms— milliseconds spent writing the flush.
Changing the sample rate, or switching the meter off
Neither one needs a restart. The client re-reads ems_control_setting.json at
least once a second and applies the change on the next cycle:
- Set
device.poll_interval_msto the new value (10–3000 ms). - Optionally set
device.save_interval_ms;save_interval_ms / poll_interval_msis what becomes the row size.
The reload is logged with the fields that changed:
[INFO] settings reloaded: poll_interval_ms=100, save_interval_ms=180000
[INFO] frequency meter is now active; polling and saving resume
Values that change what a sample means — datatype, endianness, model,
device_id, gateway_id — are reported and then ignored, because applying them
mid-window would mix two meanings into one row. Those still need the restart.
If the interval changes, revisit save_interval_ms, otherwise the points drawn
per chart will drift.
Changing the model
The client resolves the model once at startup, so a model change restarts the client — the same step as above:
- Set the model for
frequencyMeter_register.jsoninemsConfig/selected_model.json(either the globalmodelor aby_register_fileoverride). - Restart the client by restarting
ems-backend. - Apply the matching change on the simulator side and restart it, so both sides
serve and read the same register layout. The simulator has the same
startup-resolution constraint; see the simulator repository's
docs/model_selection.mdfor its selection file and restart step.
Because the model fixes the register addresses, a mismatch does not degrade
gracefully: the client reads the wrong addresses or the server answers
IllegalAddress, and the chart stays empty.
Verification
# backend rows
docker exec mariadb mysql -u scada_user -p -N -B -e \
"SELECT COUNT(*) FROM ems_backend_db.frequency_meter_measurements;"
# frontend rows, keyed correctly
docker exec mariadb mysql -u scada_user -p -N -B -e \
"SELECT component_type, gateway_id, device_id, COUNT(*)
FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;"
Expected identity in the second query: frequency_meters | GTW-ems100CX |
FREQUENCY-METER-001. Any other gateway_id means identity resolution is
misconfigured.