BMU_Client_Save_Data
The BMU Modbus TCP Client with TimescaleDB Integration (bmu_client_save_data_v6.py) is a Python-based data acquisition and storage tool designed for Battery Management Unit (BMU) monitoring within an Energy Management System (EMS).
This script establishes a Modbus TCP connection to one or more BMUs, reads measurement registers as defined in configurable JSON register maps, and performs the following core tasks:
Real-Time Data Acquisition
- Connects to BMU devices via Modbus TCP.
- Reads register values (voltages, temperatures, status flags, etc.) from each BMU according to its configuration.
Data Processing & Normalization
- Flattens nested register data into a simplified JSON structure.
- Normalizes units (e.g., converts centivolts to volts, deci-degrees to °C).
Data Storage
- Saves each BMU's readings as timestamped JSON files in the
bmuData/directory, the BMU key in the filename. - Stores the latest snapshot for each BMU in the lastValue/ folder for quick access.
- Inserts individual data points into a TimescaleDB hypertable (bmu_measurements) for time-series analysis.
Database Management
- Automatically creates the required TimescaleDB database and table if they do not exist.
Configuration & Flexibility
- Uses external JSON files for Modbus connection settings (bms_modbus_config.json), BMU configuration (bmu_config.json), and BMU register maps (bmu_register_*.json).
- Adjustable polling and saving intervals via configuration files.
Usability Features
- Built-in --help argument for quick usage guidance.
- Error handling for file saving, Modbus communication, and database operations.
Intended Use:
- This script is intended for continuous BMU monitoring in stationary battery systems, enabling real-time performance tracking, fault detection, and historical data analysis in integration with SCADA and EMS platforms.
⚙️ File Tree
project-root/
├── bmu_client_save_data_v6.py
├── bmsData/
│ └── bmuX/ # Output JSON data per BMU (X = 1, 2, etc.)
├── lastValue/ # Latest snapshot JSON for each BMU
├── emsConfig/
│ ├── .env # Database credentials (DB_NAME, DB_USER, DB_PASSWORD, DB_HOST, DB_PORT)
│ ├── bms_modbus_config.json # Modbus TCP connection settings + polling/saving intervals
│ └── bmu/
│ ├── bmu_config.json # BMU count + register file mapping
│ ├── bmu_register_1.json # Register map for BMU1
│ ├── bmu_register_2.json # Register map for BMU2 (optional)
│ └── ... # Additional BMU register files as needed
Note
- bmu_client_save_data_v6.py reads from emsConfig/ and writes to bmsData/ and lastValue/.
- The bms_modbus_config.json file defines Modbus connection details and timing parameters.
- The bmu_config.json file defines how many BMUs exist and which register file each uses.
- Each bmu_register_X.json defines the register addresses and categories for that BMU.
⚙️ File Map
+------------------------------------------------------+
| bmu_client_save_data_v6.py |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| emsConfig/.env [REQUIRED] |
| • Database name, user, password, host, port |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| emsConfig/bms_modbus_config.json [REQUIRED] |
| • Modbus server IP and port |
| • polling_interval_ms |
| • saving_interval_s |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| emsConfig/bmu/bmu_config.json [REQUIRED] |
| • number_of_bmu |
| • bmuX -> register file mapping |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| emsConfig/bmu/bmu_register_X.json [REQUIRED] |
| • Register categories: read_only, write_only, etc. |
| • name -> address mapping |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| bmsData/bmuX/<device_id>_data_<timestamp>.json |
| [OUTPUT] |
| • Flattened BMU data (voltage, temp, alarms, etc.) |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| lastValue/latest_bmuX.json [OUTPUT] |
| • Latest snapshot for quick access |
+------------------------------------------------------+
|
v
+------------------------------------------------------+
| TimescaleDB (bmu_measurements table) [OUTPUT] |
| • Time-series storage of BMU values |
+------------------------------------------------------+
.env file
DB_NAME=ems_local
DB_USER=postgres
DB_PASSWORD=<password>
DB_HOST=localhost
DB_PORT=5432
Python Code
bcms_client_save_data_v6.py
#!/usr/bin/env python3
"""
BMU Modbus TCP Client with TimescaleDB Integration
- Reads BMU data over Modbus TCP
- Flattens and saves JSON using device_id
- Inserts each value to TimescaleDB (bmu_measurements table)
- v6 save the last values to the lastValue folder
"""
import os
import time
import json
import sys
import psycopg2
from datetime import datetime
from pyModbusTCP.client import ModbusClient
from dotenv import load_dotenv
from pathlib import Path
import os
import json
from datetime import datetime
# New constant
LATEST_OUTPUT_DIR = "lastValue"
# === Config paths ===
MODBUS_CONFIG = "emsConfig/bcms_modbus_config.json"
BMU_CONFIG = "emsConfig/bmu/bmu_config.json"
BMU_DIR = "emsConfig/bmu"
OUT_ROOT = "bcmsData"
ENV_PATH = "emsConfig/.env"
# === Help text ===
HELP_TEXT = """
BMU Modbus Client + TimescaleDB Writer
USAGE:
python3 bmu_client_save_data_v1.py [--help]
DESCRIPTION:
- Connects to Modbus TCP server to read BMU registers
- Saves JSON per BMU using device_id (flattened format)
- Inserts data into TimescaleDB (table: bmu_measurements)
OUTPUT FILE FORMAT:
bcmsData/bmuX/<device_id>_data_<timestamp>.json
DEPENDENCIES:
pip install pyModbusTCP psycopg2-binary python-dotenv
FILES REQUIRED:
- emsConfig/.env (DB connection)
- emsConfig/bcms_modbus_config.json
- emsConfig/bmu/bmu_config.json
- emsConfig/bmu/bmu_register_*.json
"""
if "--help" in sys.argv:
print(HELP_TEXT)
sys.exit(0)
# === Load .env ===
load_dotenv(dotenv_path=ENV_PATH)
DB_CONFIG = {
"dbname": os.getenv("DB_NAME"),
"user": os.getenv("DB_USER"),
"password": os.getenv("DB_PASSWORD"),
"host": os.getenv("DB_HOST"),
"port": os.getenv("DB_PORT", 5432)
}
ADMIN_DB = {**DB_CONFIG, "dbname": "postgres"}
# === Helpers ===
def save_latest_bmu_json(bmu_key: str, flat_data: dict):
"""
Save the latest BMU data snapshot into a 'latest_<bmu_key>.json' file in 'lastValue' directory.
Args:
bmu_key (str): The BMU identifier (e.g., 'bmu1', 'bmu2').
flat_data (dict): The most recent flattened BMU data dictionary.
"""
os.makedirs(LATEST_OUTPUT_DIR, exist_ok=True)
filename = f"latest_{bmu_key}.json"
filepath = os.path.join(LATEST_OUTPUT_DIR, filename)
try:
with open(filepath, "w") as f:
json.dump(flat_data, f, indent=2)
print(f"[✓] Latest snapshot saved: {filepath}")
except Exception as e:
print(f"[!] Failed to save latest BMU file for {bmu_key}: {e}")
def load_json(path):
with open(path, "r") as f:
return json.load(f)
def save_output(bmu_key: str, device_id: str, data: dict):
"""
Save normalized BMU output as JSON using device_id in filename and
save the latest data snapshot in a separate folder.
"""
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
folder = os.path.join(OUT_ROOT, bmu_key)
os.makedirs(folder, exist_ok=True)
filename = f"{device_id}_data_{timestamp}.json"
filepath = os.path.join(folder, filename)
flat_data = {}
for key, val in data.items():
if key == "timestamp":
flat_data["timestamp"] = val
elif key.endswith("_voltage_v") and key.startswith("cell"):
flat_data[key] = round(val / 100, 2)
elif key.endswith("_temp_c") and key.startswith("cell"):
flat_data[key] = round(val / 10, 1)
else:
flat_data[key] = val
try:
with open(filepath, "w") as f:
json.dump(flat_data, f, indent=2)
print(f"[✓] Saved: {filepath}")
except Exception as e:
print(f"[!] Failed to save data: {e}")
# Also save latest snapshot
save_latest_bmu_json(bmu_key, flat_data)
def read_single_register(client, addr: int):
regs = client.read_holding_registers(addr, 1)
return regs[0] if regs else None
def ensure_database():
"""Create database and bmu_measurements table if not exist."""
try:
conn = psycopg2.connect(**ADMIN_DB)
conn.autocommit = True
cur = conn.cursor()
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (DB_CONFIG["dbname"],))
if not cur.fetchone():
cur.execute(f"CREATE DATABASE {DB_CONFIG['dbname']}")
print(f"[DB] Created database {DB_CONFIG['dbname']}")
cur.close()
conn.close()
except Exception as e:
print(f"[!] DB creation error: {e}")
return
try:
conn = psycopg2.connect(**DB_CONFIG)
cur = conn.cursor()
cur.execute("""
CREATE TABLE IF NOT EXISTS bmu_measurements (
bmu_timestamp TIMESTAMPTZ NOT NULL,
bmu_key TEXT,
device_id TEXT,
category TEXT,
name TEXT,
value FLOAT
);
""")
cur.execute("SELECT create_hypertable('bmu_measurements', 'bmu_timestamp', if_not_exists => TRUE);")
conn.commit()
cur.close()
conn.close()
print("[DB] Table bmu_measurements ready.")
except Exception as e:
print(f"[!] Table setup failed: {e}")
def insert_bmu_data(bmu_key: str, device_id: str, data: dict):
"""Insert flattened data into TimescaleDB."""
try:
conn = psycopg2.connect(**DB_CONFIG)
cur = conn.cursor()
ts = datetime.fromtimestamp(data["timestamp"] / 1000.0)
for key, val in data.items():
if key == "timestamp":
continue
if isinstance(val, dict):
for name, v in val.items():
cur.execute("""
INSERT INTO bmu_measurements (bmu_timestamp, bmu_key, device_id, category, name, value)
VALUES (%s, %s, %s, %s, %s, %s)
""", (ts, bmu_key, device_id, "", name, v))
else:
cur.execute("""
INSERT INTO bmu_measurements (bmu_timestamp, bmu_key, device_id, category, name, value)
VALUES (%s, %s, %s, %s, %s, %s)
""", (ts, bmu_key, device_id, "", key, val))
conn.commit()
cur.close()
conn.close()
print(f"[DB] Inserted {device_id} data at {ts}")
except Exception as e:
print(f"[!] DB insert failed: {e}")
# === Main loop ===
def main():
modbus_cfg = load_json(MODBUS_CONFIG)
host = modbus_cfg.get("ip", "127.0.0.1")
port = modbus_cfg.get("port", 1507)
save_interval = modbus_cfg.get("saving_interval_s", 3)
poll_interval = modbus_cfg.get("polling_interval_ms", 2000) / 1000.0
ensure_database()
config = load_json(BMU_CONFIG)
client = ModbusClient(host=host, port=port, auto_open=True)
if not client.open():
print(f"[✗] Could not connect to {host}:{port}")
return
print(f"[✓] Connected to Modbus at {host}:{port}")
print(f"Polling every {poll_interval}s | Saving every {save_interval}s")
try:
while True:
for i in range(1, config["number_of_bmu"] + 1):
bmu_key = f"bmu{i}"
reg_file = config.get(bmu_key)
if not reg_file:
continue
reg_path = os.path.join(BMU_DIR, reg_file)
if not os.path.exists(reg_path):
continue
reg_data = load_json(reg_path)
device_id = reg_data.get("device_id", bmu_key)
categories = reg_data.get("categories", {})
entry = {
"timestamp": int(time.time() * 1000)
}
for cat_name, cat in categories.items():
for name, addr in cat.get("read_only", {}).items():
val = read_single_register(client, addr)
entry[name] = val # Flattened directly
save_output(bmu_key, device_id, entry)
insert_bmu_data(bmu_key, device_id, entry)
time.sleep(save_interval)
except KeyboardInterrupt:
print("\n[✗] Interrupted by user.")
finally:
client.close()
if __name__ == "__main__":
main()
Required Json files
bmu_config.json
Cl1_P1_data_20250809_022241.json
{
"timestamp": 1754698961200,
"cell1_voltage_v": 2.22,
"cell2_voltage_v": 3.46,
"cell3_voltage_v": 3.44,
"cell4_voltage_v": 3.47,
"cell5_voltage_v": 3.43,
"cell6_voltage_v": 3.45,
"cell7_voltage_v": 3.46,
"cell8_voltage_v": 3.45,
"cell9_voltage_v": 3.44,
"cell10_voltage_v": 3.47,
"cell11_voltage_v": 3.46,
"cell12_voltage_v": 3.44,
"cell13_voltage_v": 3.43,
"cell14_voltage_v": 3.45,
"cell15_voltage_v": 3.44,
"cell16_voltage_v": 3.46,
"cell17_voltage_v": 3.45,
"cell18_voltage_v": 3.47,
"cell19_voltage_v": 3.46,
"cell20_voltage_v": 3.45,
"cell21_voltage_v": 3.44,
"cell22_voltage_v": 3.46,
"cell23_voltage_v": 3.45,
"cell24_voltage_v": 3.44,
"cell1_temp_c": 25.0,
"cell2_temp_c": 25.3,
"cell3_temp_c": 25.2,
"cell4_temp_c": 25.1,
"cell5_temp_c": 25.0,
"cell6_temp_c": 24.9,
"cell7_temp_c": 24.8,
"cell8_temp_c": 0.0,
"cell10_temp_c": 25.4,
"cell11_temp_c": 25.2,
"cell12_temp_c": 25.0,
"cell13_temp_c": 25.3,
"cell14_temp_c": 25.1,
"cell15_temp_c": 25.2,
"cell16_temp_c": 24.9,
"cell17_temp_c": 24.8,
"cell18_temp_c": 24.7,
"cell19_temp_c": 25.0,
"cell20_temp_c": 25.3,
"cell21_temp_c": 25.4,
"cell22_temp_c": 25.2,
"cell23_temp_c": 25.0,
"cell24_temp_c": 24.8,
"pack_general_alarm": 85,
"cell_overvoltage_alarm": 90,
"cell_undervoltage_alarm": 85,
"temperature_over_alarm": 90,
"temperature_under_alarm": 55,
"communication_fault": 90,
"soc": 90,
"soh": 80
}
bmu_register_1.json
bmu_modbus_config.json
bmu_modbus_config.json