Skip to content

Frequency Meter Data Pipeline

This document traces one frequency sample from the simulator register map to a rendered point on the EMS dashboard chart. It is the reference for anyone debugging "the data is arriving but the chart is empty".

End-to-end path

flowchart TD
    S["bess-simulator-backend<br/>frequencyMeter_modbus_server<br/>TCP :1515"] -->|coalesced reads, 10 ms| C["frequencyMeter_client_save_data_v2.py<br/>Reader -> SampleBuffer -> Writer"]
    C -->|multi-row INSERT| B[("ems_backend_db<br/>frequency_meter_measurements")]
    C -->|multi-row INSERT| F[("ems_db<br/>frequency_meter_readings")]
    F --> M["App\\Models\\FrequencyMeterReading"]
    M --> K["EmsAuxiliaryDataExpandDashController<br/>buildDynamicChartSeries()"]
    K -->|POST /getEmsExpandData| J["Expanded dashboard<br/>ECharts line series"]

Stage 1 — Simulator (:1515)

The simulator serves the frequency meter register map on TCP 1515. The active map is resolved from emsConfig/selected_model.json; for frequencyMeter_register.json the shipped variants are ME231, P8000 and PM5560. See docs/model_selection.md in the simulator repository for the model-resolution rules, the by_register_file override and the restart a model switch requires — the simulator sizes its register window at startup, as the client does.

Stage 2 — Client

frequencyMeter_client_save_data_v2.py reads the map every poll_interval_ms and buffers decoded samples, minus the registers excluded by frequencyMeter_read_list.json. Setting the meter to inactive pauses polling and writing without restarting the client. See Frequency Meter Client for the full reference.

Two identities must be correct here, because everything downstream filters on them:

Field Value Resolved by
gateway_id GTW-ems100CX ems_device_identity.require_device_identity()
device_id FREQUENCY-METER-001 same
component_type frequency_meters hard-coded in the client

The most common cause of an empty chart

If gateway_id is anything other than the active gateway, the rows land in the table successfully but no query can ever return them. Writing gateway_id = "ems" (from ems_key) produces exactly this symptom. Confirm with:

SELECT component_type, gateway_id, device_id, COUNT(*)
  FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;

You want a single row: frequency_meters | GTW-ems100CX | FREQUENCY-METER-001.

Stage 3 — Two tables, one envelope

Each flush writes the same batch envelope to both databases.

Database Table Column layout Read by
ems_backend_db frequency_meter_measurements time (bigint PK), metadata (json) EMS backend services
ems_db frequency_meter_readings device_id, gateway_id, component_type, readings, site_id, … EMS frontend charts

Envelope:

{
  "batch_start_ms": 1790420751946,
  "batch_end_ms": 1790420753496,
  "sampling_interval_ms": 10,
  "batch_size": 1800,
  "sample_count": 155,
  "samples": [
    { "timestamp": 1790420751946, "frequencyMeter_frequency_hz": 50.0,
      "frequencyMeter_active_power_total_kw": 150.0, "...": 0 }
  ]
}

Why batches, not one row per sample

At 10 ms a single client produces ~8.6 million samples per day. One row per sample would make the table grow unmanageably and turn every page load into a huge scan. Batching keeps the write path efficient; the chart layer flattens the envelope back into points on read.

Stage 4 — Laravel frontend

Table and model

The table is created by the migration database/migrations/2026_09_26_000001_create_frequency_meter_readings_table.php and is modelled by App\Models\FrequencyMeterReading:

public $table = 'frequency_meter_readings';

protected $casts = [
    'id'        => 'integer',
    'site_id'   => 'integer',
    'readings'  => 'array',
];

readings is cast to an array, so $row->readings returns the decoded envelope.

Chart controller

EmsAuxiliaryDataExpandDashController::getExpandData() handles POST /getEmsExpandData. It validates the gateway, loads the component's chart_configs, and builds the series in buildDynamicChartSeries().

That method selects its source model by component name:

$sourceModel = $componentName === 'frequency_meters'
    ? FrequencyMeterReading::class
    : EmsHardwareReading::class;

Every other component keeps one flat reading per row in ems_hardware_readings; only frequency_meters routes to the batched table.

Flattening the envelope

Because the stored shape is a batch, the series builder detects a samples key and emits one [x, y] point per sample, falling back to treating the row itself as a single sample:

$rows = (isset($readingsData['samples']) && is_array($readingsData['samples']))
    ? $readingsData['samples']
    : [$readingsData];

foreach ($rows as $rowData) {
    $xValue = $rowData[$xKey] ?? null;              // 'timestamp', epoch ms -> 'Y-m-d H:i:s'
    foreach ($yDataSourceKeys as $yKey => $yLabel) {
        if (is_numeric($rowData[$yKey] ?? null)) {
            $series[$yKey]['data'][] = [$xValue, (float) $rowData[$yKey]];
        }
    }
}

Points are then capped to the most recent 2000 per series:

$maxPoints = 2000;
foreach ($series as $yKey => $ySeries) {
    if (count($ySeries['data']) > $maxPoints) {
        $series[$yKey]['data'] = array_slice($ySeries['data'], -$maxPoints);
    }
}

Do not remove the point cap without changing the row window

The query takes the latest 100 rows. At 1800 samples per row that is ~180 000 points. Sending that to the browser will stall the chart. 2000 points at 10 ms covers 20 seconds of data; raise the cap only if you also reduce the row limit or downsample server-side.

Chart configuration

The series keys come from the frequency_meters entry in RegisterCommonDeviceTableSeeder under component_details.frequency_meters.more_detail_dashboard_conifgs.chart_configs:

Setting Value
chart_id chart003
chart_type line
x key timestamp
y key frequencyMeter_active_power_total_kw

The data_visible_keys for the component uses frequencyMeter_frequency_hz.

Request contract

Parameter Value
type frequency_meters
deviceId FREQUENCY-METER-001
gatwayId GTW-ems100CX (plaintext — the decrypt call in getExpandData() is commented out)

Verifying the whole path

Work backwards from the chart.

1. Rows exist and are keyed correctly

SELECT component_type, gateway_id, device_id, COUNT(*)
  FROM ems_db.frequency_meter_readings GROUP BY 1,2,3;

2. The model sees them (inside the frontend container)

docker exec -w /var/www/html ems-frontend \
  php artisan tinker --execute="echo App\Models\FrequencyMeterReading::count();"

3. The series builder returns points

Exercise buildDynamicChartSeries() through the container so it runs against the real database and chart_configs:

$c = app(App\Http\Controllers\EmsAuxiliaryDataExpandDashController::class);
$m = new ReflectionMethod($c, 'buildDynamicChartSeries');
$m->setAccessible(true);
$res = $m->invoke($c, 'GTW-ems100CX', 'FREQUENCY-METER-001', $cfg, 'frequency_meters');

A healthy result is two series with up to 2000 points each, e.g. [2026-09-26 11:16:24, 150] → [2026-09-26 11:16:44, 150] — a 20-second span, consistent with 2000 points at 10 ms.

4. The endpoint answers

POST /getEmsExpandData should return HTTP 200 with seriesData.series[] populated. A 422 with GatewayValidity: false means the gateway argument does not match the active EmsSetting row (status_04 = 'Active'), not that there is no data.

Known gap

SensorsLatestData in the endpoint response is populated from the lastValue file mechanism, not from frequency_meter_readings. The chart line renders from the table, but the "latest snapshot" panel beside it stays empty until something feeds that path. See Last-value Templates.

Deployed components

Component Repository Path on host
Client root/es100cx12s.git /opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/
Simulator root/bess-simulator.git /opt/gess-stack/bess-simulator/backend/app/EnergySyc_EMS_ES100Cx12S_BESS_simulator/
Frontend (ems, scada, gessgo, uv) root/scada2.git /opt/gess-stack/{ems/frontend,scada,gessgo,uv}/app/

All four frontends share root/scada2.git. A frontend change belongs in that single repository; each deployment pulls it independently.