Database Backup — Laravel EMS Frontend
How to trigger a database backup from the ems-frontend Laravel app and follow
its progress while it runs.
The problem this solves: a full backup is 16 databases and takes roughly an hour,
and POST /api/v1/backup used to run the whole thing inside the request. Nothing
was observable while it ran, and a 600 s timeout on that request fired around
database 3 of 16 — killing the dump mid-write, so no UI-triggered backup could
ever finish. The API now starts the job in the background and answers at once,
and reports progress through two read-only endpoints the frontend can poll.
For the Laravel side of that — config, routes, controller methods and the browser polling loop — see Backup & Job Status API — Laravel Integration.
Architecture
flowchart LR
B[Browser] -->|login session| N[ems-nginx]
N --> F[ems-frontend<br/>Laravel php-fpm]
F -->|"POST /api/v1/backup<br/>-> 202 at once"| D[db-backup:8096<br/>GESS backup API]
F -->|"poll GET /api/v1/job/status"| D
D -->|"backup_manager.sh backup --type db"| S[(/opt/backup<br/>on /dev/sdb)]
| Component | Detail |
|---|---|
ems-frontend |
Laravel app, php-fpm only — no HTTP listener, so shell tests must target ems-nginx |
db-backup |
GESS backup API (zero-dependency Python 3), listens on 8096 |
/scripts |
Backup scripts bind-mounted read-only from /opt/gess-stack/ |
/opt/backup |
Backup storage, mounted on /dev/sdb |
| Scheduled | DB backup 02:00, folder backup 02:30, disk check every 6 h (container entrypoint) |
Prerequisites
Two credentials, both required by every EMS outbound route:
| Credential | Where it comes from | How to read it |
|---|---|---|
| EMS bearer token | server_bearer_tokens where status_01 = 'EMS' |
see below |
u_key |
users.unic_key |
see below |
docker exec -u www-data ems-frontend php artisan tinker --execute='echo DB::table("server_bearer_tokens")->where("status_01","EMS")->value("bearer_token");'
docker exec -u www-data ems-frontend php artisan tinker --execute='echo DB::table("users")->value("unic_key");'
Configuration the Laravel side uses:
// config/ems.php
'backup_api_url' => env('BACKUP_API_INTERNAL_URL', 'http://db-backup:8096'),
'backup_api_token' => env('BACKUP_ALERT_API_TOKEN'),
Routes
All four sit in the v1/ems group behind the verify.emsHardware_bearer
middleware (routes/api.php:140), so each needs the Authorization header.
| Method | Path | Controller | Purpose |
|---|---|---|---|
POST |
/api/v1/ems/trigger-backup |
triggerBackup |
Start a backup (returns at once) |
GET |
/api/v1/ems/backup-status |
backupStatus |
Progress of the running job |
GET |
/api/v1/ems/backup-health |
checkBackupHealth |
API + mount health |
POST |
/api/v1/ems/trigger-restore |
triggerRestore |
Start a restore (returns at once) |
There is also a session-authenticated debug page at /EmsOutboundDebug
(routes/web.php:2622) with POST /EmsOutboundDebug/backup and
/EmsOutboundDebug/backups behind it.
Step 1 — Trigger a database backup
type is validated as all, db or files; use db for databases only.
docker exec ems-frontend sh -lc '
TOK=$(sed -n "s/^EMS_SETTINGS_API_TOKEN=//p" .env | head -1 | tr -d "\r\"")
curl -sS -X POST http://ems-nginx/api/v1/ems/trigger-backup \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOK" \
-d "{\"type\":\"db\",\"u_key\":\"<user-u-key>\"}"'
It returns immediately with 202:
{
"status": "accepted",
"operation": "backup",
"type": "db",
"message": "backup started in the background.",
"timestamp": "2026-10-02 10:41:07",
"status_url": "/api/v1/job/status",
"log_url": "/api/v1/job/log",
"db_name": ""
}
The job then runs in a worker thread inside db-backup, so the request no longer
has to survive the whole backup. Follow it with step 2.
This response used to block and has changed shape. It was
200carrying the finished run'sstdout,exit_codeandlog_file, because it waited for the job to end. It is now202with just the acknowledgement — read the output fromGET /api/v1/job/loginstead.triggerBackup()checks$response->successful(), which covers202, so the Laravel route is unaffected; anything readingstdoutoff this response is not.
While a job runs, a second POST gets 409: backup, restore and restart are
serialized on purpose. The lock is released when the job ends, and the next
POST is accepted.
There is no request deadline any more — and the job limit is not a deadline
either. BACKUP_JOB_TIMEOUT in gess-stack.conf (default 1200, twenty minutes)
is how long a job may write nothing at all before it is stopped: no new log
lines, no growth in the .tmp a dump streams into, and nothing written under the
backup mount. A backup that keeps producing is never interrupted, however long it
takes — an hour-long full backup passes straight through.
If it does fire, the job is stopped and the dump still running inside the
database container is stopped with it — see the note on docker exec in
Operational notes. The exit code is 124.
Step 2 — Watch progress
GET /api/v1/ems/backup-status proxies the backup API's live job state. Poll it
from a second shell while step 1 is still running.
docker exec ems-frontend sh -lc '
TOK=$(sed -n "s/^EMS_SETTINGS_API_TOKEN=//p" .env | head -1 | tr -d "\r\"")
curl -sS -H "Authorization: Bearer $TOK" \
"http://ems-nginx/api/v1/ems/backup-status?u_key=<user-u-key>"'
Presentable live view
?format=text renders one line — step counter, current database, bytes written,
write rate, elapsed and ETA — and rewrites it in place, so no TERM and no
clear are needed:
docker exec ems-frontend sh -lc '
while :; do
printf "\r%s\033[K" "$(curl -sS "http://db-backup:8096/api/v1/job/status?format=text")"
curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
sleep 3
done; echo'
It prints something like:
[####............] 3/16 | databases | ems_db 6.81 GB @ 12.3 MB/s | elapsed 6m12s | ETA ~18m40s
and exits by itself as soon as the job stops running, so it does not spin
forever printing no backup or restore is running.
\033[K erases from the cursor to the end of the line, which is what keeps a
shorter update from leaving stale characters behind. A fixed %-120s pad does
not work: on a terminal narrower than 120 columns the padding wraps to a second
line and each refresh smears another copy across the screen.
Start the watcher after the backup is running. running is false both when
a job has finished and when none has started, so a watcher launched first exits
on its first iteration — usually what you want, but if you would rather wait for
the job to appear, grep for "running": true first:
docker exec ems-frontend sh -lc '
while :; do
curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" && break
sleep 2
done
while :; do
printf "\r%s\033[K" "$(curl -sS "http://db-backup:8096/api/v1/job/status?format=text")"
curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
sleep 3
done; echo'
The raw script log is still available:
docker exec ems-frontend sh -lc '
while :; do
printf "[%s] " "$(date +%H:%M:%S)"
curl -sS "http://db-backup:8096/api/v1/job/log?lines=1"
curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
sleep 3
done'
Polling
?lines=1on its own is not useful progress. The script printsBackup file: <path>once and then nothing for the whole dump — it streams into<path>.tmpand renames it at the end — so the last line stays identical for minutes at a time. Use?format=text, or watch the growing file directly on the host:ls -lh /opt/backup/database/*/*.tmp.
A full-screen variant — note the -e TERM=xterm. Without it, clear fails with
TERM environment variable not set., because docker exec starts the process
with an empty environment:
docker exec -e TERM=xterm ems-frontend sh -lc '
while :; do
clear
curl -sS "http://db-backup:8096/api/v1/job/status?format=text"
curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
sleep 3
done'
They all exit by themselves once the job stops running. Ctrl-C stops one
early; Ctrl-Z does not — it suspends the whole docker exec as a background
job, which then sits there until you kill %1.
Underlying endpoints on db-backup
| Route | Returns |
|---|---|
GET /api/v1/job/status |
JSON: job state, progress, log_tail |
GET /api/v1/job/status?format=text |
The same as one presentable line of text/plain |
GET /api/v1/job/log?lines=N |
Last N lines (default 200, max 5000) as text/plain |
GET /api/v1/job/status fields:
| Field | Notes |
|---|---|
running |
true while the backup is in flight |
operation |
backup or restore |
type |
the --type value that was requested |
pid |
PID of backup_manager.sh |
started_at / finished_at |
YYYY-MM-DD HH:MM:SS or null |
elapsed_seconds / elapsed_human |
since start; ticks live while running |
exit_code |
null while running, 0 on success |
log_file |
path of the live log inside db-backup |
progress |
step counters plus live byte throughput — see below |
log_tail |
last 15 log lines |
progress combines the step counters with the live byte throughput:
| Key | Notes |
|---|---|
phase |
starting → databases → folders → finished |
databases_done |
databases completed |
databases_started |
databases started (one may still be in flight) |
databases_total |
16, read from DEFAULT_DATABASES in backup_manager.sh |
current_database |
the database being dumped right now |
current_file |
the .tmp file that dump is streaming into |
bytes_written / bytes_written_human |
size of that file so far |
write_rate_bytes_per_s / write_rate_human |
growth rate, averaged across polls |
file_age_seconds |
since the file last changed; a large value means stalled |
eta_seconds / eta_human |
estimate, see below |
Completed databases are counted from the script's own
Backup completed: .../database/<db>/<file> lines, not guessed.
eta_seconds averages the durations of the databases that already finished in
this run — taken from the timestamps in the log — and extrapolates over the ones
still to start plus the remainder of the one in flight. It needs at least two
databases to have started, so it reads null early in a run, and it is an order
of magnitude rather than a promise: database sizes here are very uneven
(ems_db alone is ~10 GB).
Where the byte count comes from
db_manager.sh writes each dump to <file>.tmp and renames it only on success
(db_manager.sh:634-660), and prints the final path just once at the start. The
API therefore measures the size of that .tmp file — the only real byte-level
progress available. Files untouched for more than 300 s are ignored, so
leftovers from aborted runs are not mistaken for the live dump.
What the output looks like
{
"status": "ok",
"running": true,
"operation": "backup",
"type": "db",
"pid": 412,
"started_at": "2026-10-02 09:54:48",
"finished_at": null,
"elapsed_seconds": 372.1,
"elapsed_human": "6m12s",
"exit_code": null,
"log_file": "/tmp/gess-backup-jobs/backup-20261002-095448.log",
"progress": {
"phase": "databases",
"databases_done": 1,
"databases_started": 2,
"databases_total": 16,
"current_database": "ems_db",
"current_file": "/opt/backup/database/ems_db/ems_db_2026-10-02_09-54-54.sql.tmp",
"bytes_written": 7307282261,
"bytes_written_human": "6.81 GB",
"write_rate_bytes_per_s": 12897484.0,
"write_rate_human": "12.30 MB/s",
"file_age_seconds": 1.2,
"eta_seconds": 1120.4,
"eta_human": "18m40s"
},
"log_tail": ["[INFO] 2026-10-02 09:54:54 Backup file: /opt/backup/database/ems_db/ems_db_2026-10-02_09-54-54.sql"]
}
The same moment as ?format=text:
[#...............] 1/16 | databases | ems_db 6.81 GB @ 12.30 MB/s | elapsed 6m12s | ETA ~18m40s
When it finishes the same call reports running: false, exit_code: 0 and
phase: finished, and the byte fields drop out because there is no active
.tmp file any more.
Databases included in type: db
16 databases, in the order the script walks them:
scada_db ems_db ems_backend_db gessgo_db
ismo_db uv_db gridsync_db powernode_db
ai_assistant_db bess_simulator_db ev_site_db uv_site_db
ems_site_db scada_site_db gridsync_site_db powernode_site_db
type values and what each does:
type |
Script behaviour |
|---|---|
db |
databases only |
files |
storage folders only |
all |
databases, then folders (default) |
Wiring it into the Laravel app
Fire the backup, then poll — the trigger no longer waits for the job, so it can use a short timeout.
// No longer needs to outlive the backup: it returns 202 straight away.
$response = Http::timeout(30)
->acceptJson()
->post(rtrim(config('ems.backup_api_url'), '/') . '/api/v1/backup', [
'type' => 'db',
]);
// 202 Accepted; the dump itself keeps running on db-backup.
// Follow the job started above.
async function pollBackupStatus() {
const res = await fetch(`/api/v1/ems/backup-status?u_key=${uKey}`, {
headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' },
});
const { response: job } = await res.json();
const p = job.progress;
console.log(job.running
? `${p.phase} ${p.databases_done}/${p.databases_total} (${job.elapsed_seconds}s)`
: `finished, exit ${job.exit_code}`);
if (job.running) setTimeout(pollBackupStatus, 5000);
}
Checking the result
GET /api/v1/backups lists what is on the backup disk. It runs
backup_manager.sh list, so it can take a while. It returns one of two shapes:
{ "status": "success", "output": "<human-readable table>" }
{
"status": "success",
"data": {
"backup_root": "/opt/backup",
"databases": {
"scada_db": [
{
"filename": "scada_db_2026-10-02_020000.sql.gz",
"size_bytes": 4242,
"modified": "2026-10-02 02:00:11"
}
]
},
"folders": {}
}
}
GET /api/v1/health confirms storage before you start:
{
"status": "ok",
"service": "gess-backup-api",
"concurrency": { "server_class": "ThreadingHTTPServer", "threaded": true },
"backup_storage": { "mount_point": "/opt/backup", "is_mounted": true, "device": "/dev/sdb" }
}
Errors
| Symptom | Cause | Fix |
|---|---|---|
401 {"error":"Token missing"} |
No Authorization header |
Send Authorization: Bearer <ems-token> |
401 {"error":"Invalid token"} |
Token not in server_bearer_tokens with status_01='EMS' |
Use the token from that row |
401 {"message":"User unauthenticated."} |
Unknown u_key |
Pass a real users.unic_key |
409 {"error":"Another backup, restore or restart is already running."} |
MUTATION_LOCK is held |
Wait — mutations are serialized on purpose |
202 {"status":"accepted"} |
Not an error — the job is running in the background | Follow it with /api/v1/ems/backup-status (step 2) |
200 from POST /api/v1/backup with stdout in the body |
An older build that still ran the job inside the request | Update db-backup as in Deployment |
500 with "backup_manager.sh script not found" |
Scripts not mounted | Check the /scripts bind mounts on db-backup |
502 / connection error |
Wrong backup_api_url |
Confirm BACKUP_API_INTERNAL_URL resolves to http://db-backup:8096 |
TERM environment variable not set. |
clear without a terminal type |
Drop clear, or add -e TERM=xterm to docker exec |
Operational notes
- The progress log is ephemeral. It lives in
/tmp/gess-backup-jobsinsidedb-backup, so a container restart clears both the log and the in-memory job state —/api/v1/job/statussimply reads idle again. It is a progress view, not an audit trail; the durable output is the backup files themselves. - Mutations are serialized. Backup, restore and restart all take
MUTATION_LOCK. A backup or restore is detached, so the request thread acquires the lock and the worker thread releases it when the job ends; any other mutation in the meantime gets409. - The job log streams to disk, not to the response.
run_job()writes the command's combined stdout and stderr tolog_fileline by line, which is what letsPOST /api/v1/backupanswer202without losing the output. Read it fromGET /api/v1/job/log; there is nostdoutkey in the202body any more. - A stall stops the dump where it runs. A job is only stopped once it has
written nothing for
BACKUP_JOB_TIMEOUT.run_job()then SIGKILLs the job's process group, butmariadb-dumpruns inside the database container — a different PID namespace, withdockerdholding the pipe — so the group kill leaves it orphaned: still running, still holding a--single-transactionsnapshot open, writing into a pipe nobody reads, with its.tmpnever renamed. Measured withdocker exec mariadb sleep 300and a 5 s limit, it survived atPPID 0. The API therefore also issues, inside that container:
docker exec <container> pkill -f '<tool>.*<database>'
using the Container: and Database: lines the script logs just before it
starts the dump. The match is scoped to that database name, so a dump of any
other database — including one started by the 02:00 cron — is left alone.
- What counts as "still working". Telling a slow job from a stalled one is
what the 1200 s limit is for, and it is judged on writes, checked in this order:
- the job log, where the per-database and phase lines land
- the
.tmpfile a dump streams into — a dump logs nothing betweenBackup file: ...andBackup completed: ..., so this is the only thing that shows it is still alive - anything else under the backup mount, matched on ctime as well as mtime,
because
rsync -apreserves modification times: a folder backup copied today still carries the mtimes of its source files
Checks 1 and 2 are effectively free. Check 3 walks the mount and stops at the first recent entry; when nothing is recent it is a full scan, about 1 s for the ~154k entries on this disk. It is only reached when the first two say nothing, and the probe runs every 30 s. - Deploying a backup-API change (from the dev machine):
scp backup_api_server.py root@<host>:/opt/gess-stack/backup_api_server.py
ssh root@<host> 'docker restart db-backup'
/opt/gess-stack is not a git checkout, and the file is bind-mounted
read-only into db-backup, so the restart is what loads the new code. Keep a
backup_api_server.py.bak-<timestamp> copy first.
- Laravel route cache. Routes are cached in bootstrap/cache/routes-v7.php.
After pulling a change that adds a route, run inside ems-frontend:
docker exec -u www-data ems-frontend php artisan route:cache
Without it the new route 404s even though it exists on disk.
Full endpoint reference (db-backup:8096)
| Method | Path | Purpose |
|---|---|---|
GET |
/api/v1/health |
Service + backup mount status |
GET |
/api/v1/disk-usage |
Disk metrics for sda, sdb, sdc |
POST |
/api/v1/disk-check |
Evaluate disk thresholds, send alerts |
GET |
/api/v1/self-diagnostics |
Run EMS self-diagnostics |
POST |
/api/v1/self-diagnostics |
Self-diagnostics with a component filter |
GET |
/api/v1/containers |
List Docker containers and state |
POST |
/api/v1/restart/containers |
Soft-restart containers |
POST |
/api/v1/restart/hardware |
Hard restart (reboot host) |
POST |
/api/v1/restart |
Generic restart (type: soft or hard) |
GET |
/api/v1/backups |
List database and folder backups |
GET |
/api/v1/job/status |
Progress of the running backup/restore |
GET |
/api/v1/job/log |
Tail the live job log |
POST |
/api/v1/backup |
Start a backup, returns 202 at once |
POST |
/api/v1/restore |
Start a restore, returns 202 at once |