Frequency Meter Data Pipeline
This document traces one frequency sample from the simulator register map to a rendered point on the EMS dashboard chart. It is the reference for anyone debugging "the data is arriving but the chart is empty".
End-to-end path
flowchart TD
S["bess-simulator-backend<br/>frequencyMeter_modbus_server<br/>TCP :1515"] -->|coalesced reads, 10 ms| C["frequencyMeter_client_save_data_v2.py<br/>Reader -> SampleBuffer -> Writer"]
C -->|multi-row INSERT| B[("ems_backend_db<br/>frequency_meter_measurements")]
C -->|multi-row INSERT| F[("ems_db<br/>frequency_meter_readings")]
F --> M["App\\Models\\FrequencyMeterReading"]
M --> K["EmsAuxiliaryDataExpandDashController<br/>buildDynamicChartSeries()"]
K -->|POST /getEmsExpandData| J["Expanded dashboard<br/>ECharts line series"]
Stage 1 — Simulator (:1515)
The simulator serves the frequency meter register map on TCP 1515. The active
map is resolved from emsConfig/selected_model.json; for
frequencyMeter_register.json the shipped variants are ME231, P8000 and
PM5560. See docs/model_selection.md in the simulator repository for the
model-resolution rules, the by_register_file override and the restart a model
switch requires — the simulator sizes its register window at startup, as the
client does.
Stage 2 — Client
frequencyMeter_client_save_data_v2.py reads the map every
poll_interval_ms and buffers decoded samples, minus the registers excluded by
frequencyMeter_read_list.json. Setting the meter to inactive pauses polling
and writing without restarting the client. See
Frequency Meter Client for the full
reference.
Two identities must be correct here, because everything downstream filters on them:
| Field | Value | Resolved by |
|---|---|---|
gateway_id |
GTW-ems100CX |
ems_device_identity.require_device_identity() |
device_id |
FREQUENCY-METER-001 |
same |
component_type |
frequency_meters |
hard-coded in the client |
The most common cause of an empty chart
If gateway_id is anything other than the active gateway, the rows land in
the table successfully but no query can ever return them. Writing
gateway_id = "ems" (from ems_key) produces exactly this symptom. Confirm
with:
SELECT component_type, gateway_id, device_id, COUNT(*)
FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;
You want a single row: frequency_meters | GTW-ems100CX |
FREQUENCY-METER-001.
Stage 3 — Two tables, one envelope
Each flush writes the same batch envelope to both databases.
| Database | Table | Column layout | Read by |
|---|---|---|---|
ems_backend_db |
frequency_meter_measurements |
time (bigint PK), metadata (json) |
EMS backend services |
ems_db |
frequency_meter_readings |
device_id, gateway_id, component_type, readings, site_id, … |
EMS frontend charts |
Envelope:
{
"batch_start_ms": 1790420751946,
"batch_end_ms": 1790420753496,
"sampling_interval_ms": 10,
"batch_size": 1800,
"sample_count": 155,
"samples": [
{ "timestamp": 1790420751946, "frequencyMeter_frequency_hz": 50.0,
"frequencyMeter_active_power_total_kw": 150.0, "...": 0 }
]
}
Why batches, not one row per sample
At 10 ms a single client produces ~8.6 million samples per day. One row per sample would make the table grow unmanageably and turn every page load into a huge scan. Batching keeps the write path efficient; the chart layer flattens the envelope back into points on read.
Stage 4 — Laravel frontend
Table and model
The table is created by the migration
database/migrations/2026_09_26_000001_create_frequency_meter_readings_table.php
and is modelled by App\Models\FrequencyMeterReading:
public $table = 'frequency_meter_readings';
protected $casts = [
'id' => 'integer',
'site_id' => 'integer',
'readings' => 'array',
];
readings is cast to an array, so $row->readings returns the decoded envelope.
Chart controller
EmsAuxiliaryDataExpandDashController::getExpandData() handles
POST /getEmsExpandData. It validates the gateway, loads the component's
chart_configs, and builds the series in buildDynamicChartSeries().
That method selects its source model by component name:
$sourceModel = $componentName === 'frequency_meters'
? FrequencyMeterReading::class
: EmsHardwareReading::class;
Every other component keeps one flat reading per row in ems_hardware_readings;
only frequency_meters routes to the batched table.
Flattening the envelope
Because the stored shape is a batch, the series builder detects a samples key
and emits one [x, y] point per sample, falling back to treating the row itself
as a single sample:
$rows = (isset($readingsData['samples']) && is_array($readingsData['samples']))
? $readingsData['samples']
: [$readingsData];
foreach ($rows as $rowData) {
$xValue = $rowData[$xKey] ?? null; // 'timestamp', epoch ms -> 'Y-m-d H:i:s'
foreach ($yDataSourceKeys as $yKey => $yLabel) {
if (is_numeric($rowData[$yKey] ?? null)) {
$series[$yKey]['data'][] = [$xValue, (float) $rowData[$yKey]];
}
}
}
Points are then capped to the most recent 2000 per series:
$maxPoints = 2000;
foreach ($series as $yKey => $ySeries) {
if (count($ySeries['data']) > $maxPoints) {
$series[$yKey]['data'] = array_slice($ySeries['data'], -$maxPoints);
}
}
Do not remove the point cap without changing the row window
The query takes the latest 100 rows. At 1800 samples per row that is ~180 000 points. Sending that to the browser will stall the chart. 2000 points at 10 ms covers 20 seconds of data; raise the cap only if you also reduce the row limit or downsample server-side.
Chart configuration
The series keys come from the frequency_meters entry in
RegisterCommonDeviceTableSeeder under
component_details.frequency_meters.more_detail_dashboard_conifgs.chart_configs:
| Setting | Value |
|---|---|
chart_id |
chart003 |
chart_type |
line |
| x key | timestamp |
| y key | frequencyMeter_active_power_total_kw |
The data_visible_keys for the component uses frequencyMeter_frequency_hz.
Request contract
| Parameter | Value |
|---|---|
type |
frequency_meters |
deviceId |
FREQUENCY-METER-001 |
gatwayId |
GTW-ems100CX (plaintext — the decrypt call in getExpandData() is commented out) |
Verifying the whole path
Work backwards from the chart.
1. Rows exist and are keyed correctly
SELECT component_type, gateway_id, device_id, COUNT(*)
FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;
2. The model sees them (inside the frontend container)
docker exec -w /var/www/html ems-frontend \
php artisan tinker --execute="echo App\Models\FrequencyMeterReading::count();"
3. The series builder returns points
Exercise buildDynamicChartSeries() through the container so it runs against the
real database and chart_configs:
$c = app(App\Http\Controllers\EmsAuxiliaryDataExpandDashController::class);
$m = new ReflectionMethod($c, 'buildDynamicChartSeries');
$m->setAccessible(true);
$res = $m->invoke($c, 'GTW-ems100CX', 'FREQUENCY-METER-001', $cfg, 'frequency_meters');
A healthy result is two series with up to 2000 points each, e.g.
[2026-09-26 11:16:24, 150] → [2026-09-26 11:16:44, 150] — a 20-second span,
consistent with 2000 points at 10 ms.
4. The endpoint answers
POST /getEmsExpandData should return HTTP 200 with seriesData.series[]
populated. A 422 with GatewayValidity: false means the gateway argument
does not match the active EmsSetting row (status_04 = 'Active'), not that
there is no data.
Known gap
SensorsLatestData in the endpoint response is populated from the lastValue
file mechanism, not from frequency_meter_readings. The chart line renders
from the table, but the "latest snapshot" panel beside it stays empty until
something feeds that path. See
Last-value Templates.
Deployed components
| Component | Repository | Path on host |
|---|---|---|
| Client | root/es100cx12s.git |
/opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/ |
| Simulator | root/bess-simulator.git |
/opt/gess-stack/bess-simulator/backend/app/EnergySyc_EMS_ES100Cx12S_BESS_simulator/ |
| Frontend (ems, scada, gessgo, uv) | root/scada2.git |
/opt/gess-stack/{ems/frontend,scada,gessgo,uv}/app/ |
All four frontends share root/scada2.git. A frontend change belongs in that
single repository; each deployment pulls it independently.