Skip to content

Runbook: a device shows no data

How to tell a collector fault from a mapping fault when a dashboard chart is empty. Written from the pcs5 incident, where the instinct to look at the frontend was wrong for two hours — the fault was a Modbus port in ems-backend.

Companion pages:


1. The decision that matters

Ask one question first: does the stream exist in TimescaleDB?

                    chart is empty
                          |
            does raw_device_readings_ht hold rows
            for this source_model / device_id?
                    /                \
                 NO                  YES
                  |                    |
          COLLECTOR FAULT        MAPPING FAULT
          (ems-backend)          (ems-frontend)

Getting this backwards is the main time sink. An empty chart looks like a frontend problem because that is where it is drawn.

Memorable form:

  • Enabled but empty → collector.
  • Resolves but empty → mapping.

2. Step 1 — is the stream there?

docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$rows = DB::connection("timescale")->table("raw_device_readings_ht")
    ->select("source_model","device_id",DB::raw("count(*) as n"),DB::raw("max(time) as newest"))
    ->where("source_model","like","pcs%")
    ->groupBy("source_model","device_id")->get();
foreach ($rows as $r) printf("%-10s %-14s %-8s %s\n", $r->source_model, $r->device_id, $r->n, $r->newest);
';

Read two things: presence and freshness. A stream that exists but stopped an hour ago is a different problem from one that never existed.

No rows → section 3. Rows present → section 4.


3. Collector fault — ems-backend

Work in this order. Each step narrows the fault.

3.1 Is the unit enabled?

python3 -c "
import json
c=json.load(open('/opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/emsConfig/ems/ems_control_setting.json'))
pcs=c['general_settings']['pcs']
for k in sorted(pcs): print(k, pcs[k].get('status'))
"

status must be "active". ems_main.py gates collection on get_enabled_pcs(), which requires exactly that string. A disabled unit is not a bug — no mapping change will help.

3.2 Is the unit in the insert cycle?

docker exec ems-backend sh -c 'grep -oE "pcs[0-9]+" /app/logs/pcs_client_error.log | uniq | head -20'

A unit missing from the cycle is being dropped. In the pcs5 case the sequence ran ... pcs3, pcs4, pcs6, pcs7 ... — pcs5 never appeared.

3.3 Has its insert log stopped?

docker exec ems-backend sh -c 'ls -la /app/logs/pcs_insert_mariadb_pcs*.log'

Timestamps are the fastest signal. In the incident every unit read Oct 3 17:41 except:

pcs_insert_mariadb_pcs5.log   358546   Oct  2 06:15     <- a day behind

3.4 What does the Modbus log say?

docker exec ems-backend sh -c 'tail -30 /app/logs/pcs_modbus.log'

Look at the remote port. This is where pcs5 was caught:

ERROR COMMAND status=ERROR pcs_id=pcs5 remote=bess-simulator-backend:1520
  unit_id=4 action=set_power_kw error=Modbus write failed at step 1: address=0x0D57

Port 1520 is BMU5, not a PCS unit. The registered address 0x0D57 is a PCS register arriving at a BMU map, so every write failed. Confirm against the simulator's own port map:

docker exec bess-simulator-backend sh -c \
  'ps aux | grep -E "pcs|bmu" | grep -v grep'
PCS1..12  -> 1501..1512
BMU1..    -> 1516+        (1520 = BMU5)

3.5 Is the simulator even running?

docker ps --filter name=bess-simulator-backend --format '{{.Status}}  {{.Ports}}'
docker exec bess-simulator-backend sh -c 'ps aux | grep pcs5 | grep -v grep'

A running simulator on the wrong port and a missing simulator look identical from the backend's side. Check both.

3.6 Apply the fix — and restart

Configuration is read at startup. Changing a port without restarting changes nothing, and there is no warning:

docker exec ems-backend sh -c 'uptime'      # or: docker ps --format '{{.Status}}'

Up 27 hours after a config edit means the edit has not been loaded.

Restart the simulator first, so it is listening before the backend reconnects:

docker restart bess-simulator-backend
sleep 10
docker restart ems-backend

Then confirm:

docker exec ems-backend sh -c 'grep pcs5 /app/logs/pcs_client.log | tail -3'
docker exec ems-backend sh -c 'tail -3 /app/logs/pcs_insert_mariadb_pcs5.log'

Expect a Connected: pcs5 -> ... line and a fresh insert timestamp.


4. Mapping fault — ems-frontend

Only after step 1 shows rows present. Three checks, in order.

4.1 Does the service support the component?

docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$svc = app(App\Services\TimescaleReadingService::class);
printf("supports=%s\n", $svc->supports("grid_meters") ? "yes" : "no");
';

no → the component name is missing from the map and from risk_level_aliases. Case-insensitivity bridges PCS/pcs but not grid_meters/gridEnergyMeter.

4.2 Which id does the dashboard actually send?

Never assume this. Read it from the logs:

docker exec ems-frontend sh -c \
  'grep -ohE "deviceId[^,}]{0,40}" storage/logs/*.log | sort | uniq -c | sort -rn | head'
5504  deviceId": "GRID-METER-001"
2764  deviceId": "PCS-001"

Compare against the map's device_id and device_id_aliases.

4.3 Call fetch() with that exact id

This is the step that catches the most bugs, and the step most often skipped:

docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$svc = app(App\Services\TimescaleReadingService::class);
$r = $svc->fetch("grid_meters", null, "GRID-METER-001", "time", 5, null, null, true);
printf("rows=%d source_model=%s\n", $r->count(), $r->first()->source_model ?? "-");
';

Calling fetch() with no device id tests the default path, not the one the browser uses. An earlier round of checks passed on and was irrelevant to the failing case for exactly this reason.

A 0 here with rows present in 4.1 means a device id mismatch. Cross-check the label in ems_control_setting.json:

python3 -c "
import json
c=json.load(open('/opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/emsConfig/ems/ems_control_setting.json'))
for k,v in c['general_settings']['grid_meters'].items():
    print(k, v.get('device_id'), v.get('status'))
"

Then add the spelling to device_id_aliases and run php artisan config:clear.


5. Interpreting the result

After step 1 supports() fetch() with the browser's id Conclusion
no rows — — Collector fault — section 3
rows no — Component name missing from map / aliases
rows yes 0 rows Device id mismatch — add device_id_aliases
rows yes rows, wrong source_model Mirror missing, or stream order wrong
rows yes rows, correct source_model, chart still flat Column mapping — device_key_map_ems.php
rows yes rows, all correct, still empty php artisan config:clear not run

The wrong source_model case is worth its own alert. A check that asserts only "rows returned" passes while every unit serves unit 1's data. Always assert the returned source_model.


6. Reference: ems-backend logs

All under /app/logs/ in the ems-backend container.

File Use
pcs_client.log connection success per unit (Connected: pcsN -> host:port)
pcs_client_error.log which units are in the insert cycle
pcs_modbus.log Modbus read/write failures, with remote host and port
pcs_insert_mariadb_pcsN.log per-unit insert activity; timestamps show stalls
grid_meter_client*.log, solar_*, frequency_* same pattern per component

The startup banner prints the paths and the enabled-unit line:

docker logs ems-backend --tail 200 | grep -E "error_log|pcs_count"

7. Gotchas

  • No unit restart = no config change. A running process keeps the config it read at startup, silently.
  • Restart the simulator first. The backend connects at startup; if the simulator is not listening yet, the unit is skipped.
  • A running simulator can still be on the wrong port. Check the port in ps aux, not just that a process exists.
  • status before component counts. no_of_components: 12 with eleven inactive means one stream is correct, not eleven missing.
  • config:clear after any map edit, or nothing you changed takes effect.
  • Assert source_model, not row count. Row count alone cannot distinguish correct data from another unit's data.