Skip to content

Database Backup — Laravel EMS Frontend

How to trigger a database backup from the ems-frontend Laravel app and follow its progress while it runs.

The problem this solves: a full backup is 16 databases and takes roughly an hour, and POST /api/v1/backup used to run the whole thing inside the request. Nothing was observable while it ran, and a 600 s timeout on that request fired around database 3 of 16 — killing the dump mid-write, so no UI-triggered backup could ever finish. The API now starts the job in the background and answers at once, and reports progress through two read-only endpoints the frontend can poll.

For the Laravel side of that — config, routes, controller methods and the browser polling loop — see Backup & Job Status API — Laravel Integration.

Architecture

flowchart LR
    B[Browser] -->|login session| N[ems-nginx]
    N --> F[ems-frontend<br/>Laravel php-fpm]
    F -->|"POST /api/v1/backup<br/>-> 202 at once"| D[db-backup:8096<br/>GESS backup API]
    F -->|"poll GET /api/v1/job/status"| D
    D -->|"backup_manager.sh backup --type db"| S[(/opt/backup<br/>on /dev/sdb)]
Component Detail
ems-frontend Laravel app, php-fpm only — no HTTP listener, so shell tests must target ems-nginx
db-backup GESS backup API (zero-dependency Python 3), listens on 8096
/scripts Backup scripts bind-mounted read-only from /opt/gess-stack/
/opt/backup Backup storage, mounted on /dev/sdb
Scheduled DB backup 02:00, folder backup 02:30, disk check every 6 h (container entrypoint)

Prerequisites

Two credentials, both required by every EMS outbound route:

Credential Where it comes from How to read it
EMS bearer token server_bearer_tokens where status_01 = 'EMS' see below
u_key users.unic_key see below
docker exec -u www-data ems-frontend php artisan tinker --execute='echo DB::table("server_bearer_tokens")->where("status_01","EMS")->value("bearer_token");'
docker exec -u www-data ems-frontend php artisan tinker --execute='echo DB::table("users")->value("unic_key");'

Configuration the Laravel side uses:

// config/ems.php
'backup_api_url'   => env('BACKUP_API_INTERNAL_URL', 'http://db-backup:8096'),
'backup_api_token' => env('BACKUP_ALERT_API_TOKEN'),

Routes

All four sit in the v1/ems group behind the verify.emsHardware_bearer middleware (routes/api.php:140), so each needs the Authorization header.

Method Path Controller Purpose
POST /api/v1/ems/trigger-backup triggerBackup Start a backup (returns at once)
GET /api/v1/ems/backup-status backupStatus Progress of the running job
GET /api/v1/ems/backup-health checkBackupHealth API + mount health
POST /api/v1/ems/trigger-restore triggerRestore Start a restore (returns at once)

There is also a session-authenticated debug page at /EmsOutboundDebug (routes/web.php:2622) with POST /EmsOutboundDebug/backup and /EmsOutboundDebug/backups behind it.

Step 1 — Trigger a database backup

type is validated as all, db or files; use db for databases only.

docker exec ems-frontend sh -lc '
  TOK=$(sed -n "s/^EMS_SETTINGS_API_TOKEN=//p" .env | head -1 | tr -d "\r\"")
  curl -sS -X POST http://ems-nginx/api/v1/ems/trigger-backup \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $TOK" \
    -d "{\"type\":\"db\",\"u_key\":\"<user-u-key>\"}"'

It returns immediately with 202:

{
  "status": "accepted",
  "operation": "backup",
  "type": "db",
  "message": "backup started in the background.",
  "timestamp": "2026-10-02 10:41:07",
  "status_url": "/api/v1/job/status",
  "log_url": "/api/v1/job/log",
  "db_name": ""
}

The job then runs in a worker thread inside db-backup, so the request no longer has to survive the whole backup. Follow it with step 2.

This response used to block and has changed shape. It was 200 carrying the finished run's stdout, exit_code and log_file, because it waited for the job to end. It is now 202 with just the acknowledgement — read the output from GET /api/v1/job/log instead. triggerBackup() checks $response->successful(), which covers 202, so the Laravel route is unaffected; anything reading stdout off this response is not.

While a job runs, a second POST gets 409: backup, restore and restart are serialized on purpose. The lock is released when the job ends, and the next POST is accepted.

There is no request deadline any more — and the job limit is not a deadline either. BACKUP_JOB_TIMEOUT in gess-stack.conf (default 1200, twenty minutes) is how long a job may write nothing at all before it is stopped: no new log lines, no growth in the .tmp a dump streams into, and nothing written under the backup mount. A backup that keeps producing is never interrupted, however long it takes — an hour-long full backup passes straight through.

If it does fire, the job is stopped and the dump still running inside the database container is stopped with it — see the note on docker exec in Operational notes. The exit code is 124.

Step 2 — Watch progress

GET /api/v1/ems/backup-status proxies the backup API's live job state. Poll it from a second shell while step 1 is still running.

docker exec ems-frontend sh -lc '
  TOK=$(sed -n "s/^EMS_SETTINGS_API_TOKEN=//p" .env | head -1 | tr -d "\r\"")
  curl -sS -H "Authorization: Bearer $TOK" \
    "http://ems-nginx/api/v1/ems/backup-status?u_key=<user-u-key>"'

Presentable live view

?format=text renders one line — step counter, current database, bytes written, write rate, elapsed and ETA — and rewrites it in place, so no TERM and no clear are needed:

docker exec ems-frontend sh -lc '
  while :; do
    printf "\r%s\033[K" "$(curl -sS "http://db-backup:8096/api/v1/job/status?format=text")"
    curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
    sleep 3
  done; echo'

It prints something like:

[####............] 3/16 | databases | ems_db 6.81 GB @ 12.3 MB/s | elapsed 6m12s | ETA ~18m40s

and exits by itself as soon as the job stops running, so it does not spin forever printing no backup or restore is running.

\033[K erases from the cursor to the end of the line, which is what keeps a shorter update from leaving stale characters behind. A fixed %-120s pad does not work: on a terminal narrower than 120 columns the padding wraps to a second line and each refresh smears another copy across the screen.

Start the watcher after the backup is running. running is false both when a job has finished and when none has started, so a watcher launched first exits on its first iteration — usually what you want, but if you would rather wait for the job to appear, grep for "running": true first:

docker exec ems-frontend sh -lc '
  while :; do
    curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" && break
    sleep 2
  done
  while :; do
    printf "\r%s\033[K" "$(curl -sS "http://db-backup:8096/api/v1/job/status?format=text")"
    curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
    sleep 3
  done; echo'

The raw script log is still available:

docker exec ems-frontend sh -lc '
  while :; do
    printf "[%s] " "$(date +%H:%M:%S)"
    curl -sS "http://db-backup:8096/api/v1/job/log?lines=1"
    curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
    sleep 3
  done'

Polling ?lines=1 on its own is not useful progress. The script prints Backup file: <path> once and then nothing for the whole dump — it streams into <path>.tmp and renames it at the end — so the last line stays identical for minutes at a time. Use ?format=text, or watch the growing file directly on the host: ls -lh /opt/backup/database/*/*.tmp.

A full-screen variant — note the -e TERM=xterm. Without it, clear fails with TERM environment variable not set., because docker exec starts the process with an empty environment:

docker exec -e TERM=xterm ems-frontend sh -lc '
  while :; do
    clear
    curl -sS "http://db-backup:8096/api/v1/job/status?format=text"
    curl -sS "http://db-backup:8096/api/v1/job/status" | grep -q "\"running\": true" || break
    sleep 3
  done'

They all exit by themselves once the job stops running. Ctrl-C stops one early; Ctrl-Z does not — it suspends the whole docker exec as a background job, which then sits there until you kill %1.

Underlying endpoints on db-backup

Route Returns
GET /api/v1/job/status JSON: job state, progress, log_tail
GET /api/v1/job/status?format=text The same as one presentable line of text/plain
GET /api/v1/job/log?lines=N Last N lines (default 200, max 5000) as text/plain

GET /api/v1/job/status fields:

Field Notes
running true while the backup is in flight
operation backup or restore
type the --type value that was requested
pid PID of backup_manager.sh
started_at / finished_at YYYY-MM-DD HH:MM:SS or null
elapsed_seconds / elapsed_human since start; ticks live while running
exit_code null while running, 0 on success
log_file path of the live log inside db-backup
progress step counters plus live byte throughput — see below
log_tail last 15 log lines

progress combines the step counters with the live byte throughput:

Key Notes
phase starting → databases → folders → finished
databases_done databases completed
databases_started databases started (one may still be in flight)
databases_total 16, read from DEFAULT_DATABASES in backup_manager.sh
current_database the database being dumped right now
current_file the .tmp file that dump is streaming into
bytes_written / bytes_written_human size of that file so far
write_rate_bytes_per_s / write_rate_human growth rate, averaged across polls
file_age_seconds since the file last changed; a large value means stalled
eta_seconds / eta_human estimate, see below

Completed databases are counted from the script's own Backup completed: .../database/<db>/<file> lines, not guessed.

eta_seconds averages the durations of the databases that already finished in this run — taken from the timestamps in the log — and extrapolates over the ones still to start plus the remainder of the one in flight. It needs at least two databases to have started, so it reads null early in a run, and it is an order of magnitude rather than a promise: database sizes here are very uneven (ems_db alone is ~10 GB).

Where the byte count comes from

db_manager.sh writes each dump to <file>.tmp and renames it only on success (db_manager.sh:634-660), and prints the final path just once at the start. The API therefore measures the size of that .tmp file — the only real byte-level progress available. Files untouched for more than 300 s are ignored, so leftovers from aborted runs are not mistaken for the live dump.

What the output looks like

{
  "status": "ok",
  "running": true,
  "operation": "backup",
  "type": "db",
  "pid": 412,
  "started_at": "2026-10-02 09:54:48",
  "finished_at": null,
  "elapsed_seconds": 372.1,
  "elapsed_human": "6m12s",
  "exit_code": null,
  "log_file": "/tmp/gess-backup-jobs/backup-20261002-095448.log",
  "progress": {
    "phase": "databases",
    "databases_done": 1,
    "databases_started": 2,
    "databases_total": 16,
    "current_database": "ems_db",
    "current_file": "/opt/backup/database/ems_db/ems_db_2026-10-02_09-54-54.sql.tmp",
    "bytes_written": 7307282261,
    "bytes_written_human": "6.81 GB",
    "write_rate_bytes_per_s": 12897484.0,
    "write_rate_human": "12.30 MB/s",
    "file_age_seconds": 1.2,
    "eta_seconds": 1120.4,
    "eta_human": "18m40s"
  },
  "log_tail": ["[INFO] 2026-10-02 09:54:54 Backup file: /opt/backup/database/ems_db/ems_db_2026-10-02_09-54-54.sql"]
}

The same moment as ?format=text:

[#...............] 1/16 | databases | ems_db 6.81 GB @ 12.30 MB/s | elapsed 6m12s | ETA ~18m40s

When it finishes the same call reports running: false, exit_code: 0 and phase: finished, and the byte fields drop out because there is no active .tmp file any more.

Databases included in type: db

16 databases, in the order the script walks them:

scada_db            ems_db              ems_backend_db      gessgo_db
ismo_db             uv_db               gridsync_db         powernode_db
ai_assistant_db     bess_simulator_db   ev_site_db          uv_site_db
ems_site_db         scada_site_db       gridsync_site_db    powernode_site_db

type values and what each does:

type Script behaviour
db databases only
files storage folders only
all databases, then folders (default)

Wiring it into the Laravel app

Fire the backup, then poll — the trigger no longer waits for the job, so it can use a short timeout.

// No longer needs to outlive the backup: it returns 202 straight away.
$response = Http::timeout(30)
    ->acceptJson()
    ->post(rtrim(config('ems.backup_api_url'), '/') . '/api/v1/backup', [
        'type' => 'db',
    ]);
// 202 Accepted; the dump itself keeps running on db-backup.
// Follow the job started above.
async function pollBackupStatus() {
  const res = await fetch(`/api/v1/ems/backup-status?u_key=${uKey}`, {
    headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' },
  });
  const { response: job } = await res.json();
  const p = job.progress;
  console.log(job.running
    ? `${p.phase} ${p.databases_done}/${p.databases_total} (${job.elapsed_seconds}s)`
    : `finished, exit ${job.exit_code}`);
  if (job.running) setTimeout(pollBackupStatus, 5000);
}

Checking the result

GET /api/v1/backups lists what is on the backup disk. It runs backup_manager.sh list, so it can take a while. It returns one of two shapes:

{ "status": "success", "output": "<human-readable table>" }
{
  "status": "success",
  "data": {
    "backup_root": "/opt/backup",
    "databases": {
      "scada_db": [
        {
          "filename": "scada_db_2026-10-02_020000.sql.gz",
          "size_bytes": 4242,
          "modified": "2026-10-02 02:00:11"
        }
      ]
    },
    "folders": {}
  }
}

GET /api/v1/health confirms storage before you start:

{
  "status": "ok",
  "service": "gess-backup-api",
  "concurrency": { "server_class": "ThreadingHTTPServer", "threaded": true },
  "backup_storage": { "mount_point": "/opt/backup", "is_mounted": true, "device": "/dev/sdb" }
}

Errors

Symptom Cause Fix
401 {"error":"Token missing"} No Authorization header Send Authorization: Bearer <ems-token>
401 {"error":"Invalid token"} Token not in server_bearer_tokens with status_01='EMS' Use the token from that row
401 {"message":"User unauthenticated."} Unknown u_key Pass a real users.unic_key
409 {"error":"Another backup, restore or restart is already running."} MUTATION_LOCK is held Wait — mutations are serialized on purpose
202 {"status":"accepted"} Not an error — the job is running in the background Follow it with /api/v1/ems/backup-status (step 2)
200 from POST /api/v1/backup with stdout in the body An older build that still ran the job inside the request Update db-backup as in Deployment
500 with "backup_manager.sh script not found" Scripts not mounted Check the /scripts bind mounts on db-backup
502 / connection error Wrong backup_api_url Confirm BACKUP_API_INTERNAL_URL resolves to http://db-backup:8096
TERM environment variable not set. clear without a terminal type Drop clear, or add -e TERM=xterm to docker exec

Operational notes

  • The progress log is ephemeral. It lives in /tmp/gess-backup-jobs inside db-backup, so a container restart clears both the log and the in-memory job state — /api/v1/job/status simply reads idle again. It is a progress view, not an audit trail; the durable output is the backup files themselves.
  • Mutations are serialized. Backup, restore and restart all take MUTATION_LOCK. A backup or restore is detached, so the request thread acquires the lock and the worker thread releases it when the job ends; any other mutation in the meantime gets 409.
  • The job log streams to disk, not to the response. run_job() writes the command's combined stdout and stderr to log_file line by line, which is what lets POST /api/v1/backup answer 202 without losing the output. Read it from GET /api/v1/job/log; there is no stdout key in the 202 body any more.
  • A stall stops the dump where it runs. A job is only stopped once it has written nothing for BACKUP_JOB_TIMEOUT. run_job() then SIGKILLs the job's process group, but mariadb-dump runs inside the database container — a different PID namespace, with dockerd holding the pipe — so the group kill leaves it orphaned: still running, still holding a --single-transaction snapshot open, writing into a pipe nobody reads, with its .tmp never renamed. Measured with docker exec mariadb sleep 300 and a 5 s limit, it survived at PPID 0. The API therefore also issues, inside that container:
docker exec <container> pkill -f '<tool>.*<database>'

using the Container: and Database: lines the script logs just before it starts the dump. The match is scoped to that database name, so a dump of any other database — including one started by the 02:00 cron — is left alone. - What counts as "still working". Telling a slow job from a stalled one is what the 1200 s limit is for, and it is judged on writes, checked in this order:

  1. the job log, where the per-database and phase lines land
  2. the .tmp file a dump streams into — a dump logs nothing between Backup file: ... and Backup completed: ..., so this is the only thing that shows it is still alive
  3. anything else under the backup mount, matched on ctime as well as mtime, because rsync -a preserves modification times: a folder backup copied today still carries the mtimes of its source files

Checks 1 and 2 are effectively free. Check 3 walks the mount and stops at the first recent entry; when nothing is recent it is a full scan, about 1 s for the ~154k entries on this disk. It is only reached when the first two say nothing, and the probe runs every 30 s. - Deploying a backup-API change (from the dev machine):

scp backup_api_server.py root@<host>:/opt/gess-stack/backup_api_server.py
ssh root@<host> 'docker restart db-backup'

/opt/gess-stack is not a git checkout, and the file is bind-mounted read-only into db-backup, so the restart is what loads the new code. Keep a backup_api_server.py.bak-<timestamp> copy first. - Laravel route cache. Routes are cached in bootstrap/cache/routes-v7.php. After pulling a change that adds a route, run inside ems-frontend:

docker exec -u www-data ems-frontend php artisan route:cache

Without it the new route 404s even though it exists on disk.

Full endpoint reference (db-backup:8096)

Method Path Purpose
GET /api/v1/health Service + backup mount status
GET /api/v1/disk-usage Disk metrics for sda, sdb, sdc
POST /api/v1/disk-check Evaluate disk thresholds, send alerts
GET /api/v1/self-diagnostics Run EMS self-diagnostics
POST /api/v1/self-diagnostics Self-diagnostics with a component filter
GET /api/v1/containers List Docker containers and state
POST /api/v1/restart/containers Soft-restart containers
POST /api/v1/restart/hardware Hard restart (reboot host)
POST /api/v1/restart Generic restart (type: soft or hard)
GET /api/v1/backups List database and folder backups
GET /api/v1/job/status Progress of the running backup/restore
GET /api/v1/job/log Tail the live job log
POST /api/v1/backup Start a backup, returns 202 at once
POST /api/v1/restore Start a restore, returns 202 at once