Runbook: a device shows no data
How to tell a collector fault from a mapping fault when a dashboard chart
is empty. Written from the pcs5 incident, where the instinct to look at the
frontend was wrong for two hours — the fault was a Modbus port in ems-backend.
Companion pages:
1. The decision that matters
Ask one question first: does the stream exist in TimescaleDB?
chart is empty
|
does raw_device_readings_ht hold rows
for this source_model / device_id?
/ \
NO YES
| |
COLLECTOR FAULT MAPPING FAULT
(ems-backend) (ems-frontend)
Getting this backwards is the main time sink. An empty chart looks like a frontend problem because that is where it is drawn.
Memorable form:
- Enabled but empty → collector.
- Resolves but empty → mapping.
2. Step 1 — is the stream there?
docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$rows = DB::connection("timescale")->table("raw_device_readings_ht")
->select("source_model","device_id",DB::raw("count(*) as n"),DB::raw("max(time) as newest"))
->where("source_model","like","pcs%")
->groupBy("source_model","device_id")->get();
foreach ($rows as $r) printf("%-10s %-14s %-8s %s\n", $r->source_model, $r->device_id, $r->n, $r->newest);
';
Read two things: presence and freshness. A stream that exists but stopped an hour ago is a different problem from one that never existed.
No rows → section 3. Rows present → section 4.
3. Collector fault — ems-backend
Work in this order. Each step narrows the fault.
3.1 Is the unit enabled?
python3 -c "
import json
c=json.load(open('/opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/emsConfig/ems/ems_control_setting.json'))
pcs=c['general_settings']['pcs']
for k in sorted(pcs): print(k, pcs[k].get('status'))
"
status must be "active". ems_main.py gates collection on
get_enabled_pcs(), which requires exactly that string. A disabled unit is
not a bug — no mapping change will help.
3.2 Is the unit in the insert cycle?
docker exec ems-backend sh -c 'grep -oE "pcs[0-9]+" /app/logs/pcs_client_error.log | uniq | head -20'
A unit missing from the cycle is being dropped. In the pcs5 case the sequence
ran ... pcs3, pcs4, pcs6, pcs7 ... — pcs5 never appeared.
3.3 Has its insert log stopped?
docker exec ems-backend sh -c 'ls -la /app/logs/pcs_insert_mariadb_pcs*.log'
Timestamps are the fastest signal. In the incident every unit read Oct 3 17:41
except:
pcs_insert_mariadb_pcs5.log 358546 Oct 2 06:15 <- a day behind
3.4 What does the Modbus log say?
docker exec ems-backend sh -c 'tail -30 /app/logs/pcs_modbus.log'
Look at the remote port. This is where pcs5 was caught:
ERROR COMMAND status=ERROR pcs_id=pcs5 remote=bess-simulator-backend:1520
unit_id=4 action=set_power_kw error=Modbus write failed at step 1: address=0x0D57
Port 1520 is BMU5, not a PCS unit. The registered address 0x0D57 is a PCS
register arriving at a BMU map, so every write failed. Confirm against the
simulator's own port map:
docker exec bess-simulator-backend sh -c \
'ps aux | grep -E "pcs|bmu" | grep -v grep'
PCS1..12 -> 1501..1512
BMU1.. -> 1516+ (1520 = BMU5)
3.5 Is the simulator even running?
docker ps --filter name=bess-simulator-backend --format '{{.Status}} {{.Ports}}'
docker exec bess-simulator-backend sh -c 'ps aux | grep pcs5 | grep -v grep'
A running simulator on the wrong port and a missing simulator look identical from the backend's side. Check both.
3.6 Apply the fix — and restart
Configuration is read at startup. Changing a port without restarting changes nothing, and there is no warning:
docker exec ems-backend sh -c 'uptime' # or: docker ps --format '{{.Status}}'
Up 27 hours after a config edit means the edit has not been loaded.
Restart the simulator first, so it is listening before the backend reconnects:
docker restart bess-simulator-backend
sleep 10
docker restart ems-backend
Then confirm:
docker exec ems-backend sh -c 'grep pcs5 /app/logs/pcs_client.log | tail -3'
docker exec ems-backend sh -c 'tail -3 /app/logs/pcs_insert_mariadb_pcs5.log'
Expect a Connected: pcs5 -> ... line and a fresh insert timestamp.
4. Mapping fault — ems-frontend
Only after step 1 shows rows present. Three checks, in order.
4.1 Does the service support the component?
docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$svc = app(App\Services\TimescaleReadingService::class);
printf("supports=%s\n", $svc->supports("grid_meters") ? "yes" : "no");
';
no → the component name is missing from the map and from
risk_level_aliases. Case-insensitivity bridges PCS/pcs but not
grid_meters/gridEnergyMeter.
4.2 Which id does the dashboard actually send?
Never assume this. Read it from the logs:
docker exec ems-frontend sh -c \
'grep -ohE "deviceId[^,}]{0,40}" storage/logs/*.log | sort | uniq -c | sort -rn | head'
5504 deviceId": "GRID-METER-001"
2764 deviceId": "PCS-001"
Compare against the map's device_id and device_id_aliases.
4.3 Call fetch() with that exact id
This is the step that catches the most bugs, and the step most often skipped:
docker exec ems-frontend php -r '
require "/var/www/html/vendor/autoload.php"; $a=require "/var/www/html/bootstrap/app.php";
$a->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
$svc = app(App\Services\TimescaleReadingService::class);
$r = $svc->fetch("grid_meters", null, "GRID-METER-001", "time", 5, null, null, true);
printf("rows=%d source_model=%s\n", $r->count(), $r->first()->source_model ?? "-");
';
Calling fetch() with no device id tests the default path, not the one the
browser uses. An earlier round of checks passed on and was irrelevant to the
failing case for exactly this reason.
A 0 here with rows present in 4.1 means a device id mismatch. Cross-check the
label in ems_control_setting.json:
python3 -c "
import json
c=json.load(open('/opt/gess-stack/ems/backend/app/EnergySyc_EMS_ES100Cx12S/emsConfig/ems/ems_control_setting.json'))
for k,v in c['general_settings']['grid_meters'].items():
print(k, v.get('device_id'), v.get('status'))
"
Then add the spelling to device_id_aliases and run php artisan config:clear.
5. Interpreting the result
| After step 1 | supports() |
fetch() with the browser's id |
Conclusion |
|---|---|---|---|
| no rows | — | — | Collector fault — section 3 |
| rows | no | — | Component name missing from map / aliases |
| rows | yes | 0 rows | Device id mismatch — add device_id_aliases |
| rows | yes | rows, wrong source_model |
Mirror missing, or stream order wrong |
| rows | yes | rows, correct source_model, chart still flat |
Column mapping — device_key_map_ems.php |
| rows | yes | rows, all correct, still empty | php artisan config:clear not run |
The wrong source_model case is worth its own alert. A check that asserts
only "rows returned" passes while every unit serves unit 1's data. Always assert
the returned source_model.
6. Reference: ems-backend logs
All under /app/logs/ in the ems-backend container.
| File | Use |
|---|---|
pcs_client.log |
connection success per unit (Connected: pcsN -> host:port) |
pcs_client_error.log |
which units are in the insert cycle |
pcs_modbus.log |
Modbus read/write failures, with remote host and port |
pcs_insert_mariadb_pcsN.log |
per-unit insert activity; timestamps show stalls |
grid_meter_client*.log, solar_*, frequency_* |
same pattern per component |
The startup banner prints the paths and the enabled-unit line:
docker logs ems-backend --tail 200 | grep -E "error_log|pcs_count"
7. Gotchas
- No unit restart = no config change. A running process keeps the config it read at startup, silently.
- Restart the simulator first. The backend connects at startup; if the simulator is not listening yet, the unit is skipped.
- A running simulator can still be on the wrong port. Check the port in
ps aux, not just that a process exists. statusbefore component counts.no_of_components: 12with eleveninactivemeans one stream is correct, not eleven missing.config:clearafter any map edit, or nothing you changed takes effect.- Assert
source_model, not row count. Row count alone cannot distinguish correct data from another unit's data.