From 8cc04ea4092fad2cc5a5cb7138c66e887ed2f3d1 Mon Sep 17 00:00:00 2001 From: Eddy de Vink Date: Tue, 11 Aug 2026 20:23:24 +0200 Subject: [PATCH] feat: harden dashcam transfer tegen flakke SD-lezer + stale-mount cleanup - dashcam-transfer.py: log echte OSError (shutil.copy2 i.p.v. cp), pre-flight leesbaarheidscheck voor ruimte-wissen, retry (3x) + early-abort (5x), guards op bestandssysteem-calls, WhatsApp GESTOPT bij defecte kaart - dashcam-transfer-mount.sh: skip als service al draait, read-only fsck.exfat health check, %k device-naam doorgeven - dashcam-transfer-cleanup.sh: nieuw - ruimt stale mount op bij udev REMOVE - udev/99-dashcam-trigger.rules: ADD + REMOVE regels met %k - tests: 20 python + 7 mount + 12 cleanup checks - README: architectuur, robuustheid, deploy-instructies --- .gitignore | 4 + README.md | 65 ++++++++- dashcam-transfer-cleanup.sh | 39 +++++ dashcam-transfer-mount.sh | 28 +++- dashcam-transfer.py | 240 ++++++++++++++++++++++--------- tests/test_cleanup_script.sh | 118 ++++++++++++++++ tests/test_dashcam_transfer.py | 251 +++++++++++++++++++++++++++++++++ tests/test_mount_script.sh | 98 +++++++++++++ udev/99-dashcam-trigger.rules | 12 ++ 9 files changed, 779 insertions(+), 76 deletions(-) create mode 100755 dashcam-transfer-cleanup.sh create mode 100755 tests/test_cleanup_script.sh create mode 100644 tests/test_dashcam_transfer.py create mode 100755 tests/test_mount_script.sh create mode 100644 udev/99-dashcam-trigger.rules diff --git a/.gitignore b/.gitignore index 8182bc4..466631b 100644 --- a/.gitignore +++ b/.gitignore @@ -19,6 +19,10 @@ build/ .env.test.local .env.production.local +# Python +__pycache__/ +*.pyc + # Logs logs/ *.log diff --git a/README.md b/README.md index 41b4513..22a6598 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,51 @@ organisatie per datum en camera-positie (front/back). Draait op de **nextcloud** SD-kaart insteken → udev detecteert vendor:product (14cd:1212 of 05e3:0751) → systemd-run start /usr/local/bin/dashcam-transfer-mount %k + → skip als de transfer-service al draait (flaky lezer = geen 2e mount) → wacht tot /dev/sdX1 verschijnt (max 60s) → mount apparaat dynamisch op /mnt/sdcard (uid=1000, exfat) + → read-only fsck.exfat health check (logt, blokkeert niet) → systemctl start dashcam-transfer.service - → /usr/local/bin/dashcam-transfer.py + → /usr/local/bin/dashcam-transfer 1. Check mounts (/mnt/sdcard, /mnt/ultratouch) 2. Scan /mnt/sdcard/Novatek/Movie/ op *.MP4 - 3. Check vrije ruimte — verwijder oudste datum-mappen indien nodig - 4. Copy + verify + delete per bestand - 5. Stuur WhatsApp notificatie via wwebjs-api + 3. Leesbaarheidscheck bron (voorkomt ruimte wissen bij defecte kaart) + 4. Check vrije ruimte — verwijder oudste datum-mappen indien nodig + 5. Copy + verify + delete per bestand, met retry + 6. Early-abort bij aanhoudende I/O-fouten (SD-kaart defect) + 7. Stuur WhatsApp notificatie via wwebjs-api + +SD-kaart uittrekken + → udev REMOVE trigger (zelfde vendor:product) + → systemd-run start /usr/local/bin/dashcam-transfer-cleanup (%k) + → stale mount? (device-node weg of source = verwijderd device) + → stop dashcam-transfer.service + → umount /mnt/sdcard (fallback: umount -l) ``` +## Robuustheid + +Het script is ontworpen tegen de hierboven geziene productie-fout (defecte/flakke +SD-kaartlezer die midden in de transfer offline klapt, `I/O error`, `device offline`): + +- **Echte fout zichtbaar** — kopieerfouten worden als `OSError` met reden gelogd + (i.p.v. alleen "exit status 1"); de eerdere `cp`-stderr ging verloren. +- **Pre-flight leesbaarheidscheck** — voordat oude datum-mappen op de SSD worden + verwijderd, wordt eerst een bestand van de kaart gelezen. Een onleesbare kaart + resulteert in een duidelijke WhatsApp-"GESTOPT" en **niets** wordt gewist. +- **Retry per bestand** — 3 pogingen met 2s pauze; vangt transiënte USB-glitches. +- **Early-abort** — bij 5 opeenvolgende fouten stopt het script en stuurt het een + WhatsApp met de laatste fout. Bronbestanden worden nooit verwijderd zonder + geverifieerde kopie. +- **Geen crash mid-loop** — alle bestandssysteem-calls zijn afgeschermd. +- **fsck.exfat** — na mount draait het mount-script een read-only health check; + een dirty/defecte kaart wordt daarmee zichtbaar in syslog. +- **Geen dubbele runs** — het mount-script skip als de transfer al actief is. +- **Stale-mount opruiming** — als de kaartlezer wordt uitgetrokken (of de kaart + offline klapt) ruimt een udev REMOVE-trigger de mount op: de service wordt + gestopt en `/mnt/sdcard` netjes ontkoppeld (lazy als dat mislukt). Zo blijft er + nooit een dood device in de `mount`-output hangen. + ## Hardware | Component | Mount | Device | Doel | @@ -82,15 +116,25 @@ De container: `docker-compose.yml` in `/home/aiaiaiaiagent/wwebjs-api/`. ```udev ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="14cd", ENV{ID_MODEL_ID}=="1212", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-mount %k" ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="05e3", ENV{ID_MODEL_ID}=="0751", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-mount %k" +ACTION=="remove", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="14cd", ENV{ID_MODEL_ID}=="1212", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-cleanup %k" +ACTION=="remove", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="05e3", ENV{ID_MODEL_ID}=="0751", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-cleanup %k" ``` -`%k` geeft het kernel device (b.v. `sde1`) door aan het mount script. +`%k` geeft het kernel device (b.v. `sde1`) door aan het mount- en cleanup-script, +zodat de kaart dynamisch gemount wordt ongeacht het filesystem UUID. Na wijziging: +`sudo udevadm control --reload-rules`. ### Mount script (`/usr/local/bin/dashcam-transfer-mount.sh`) Wacht tot het doorgegeven device beschikbaar is, unmount eventuele stale mounts, mount de SD-kaart en start de service. +### Cleanup script (`/usr/local/bin/dashcam-transfer-cleanup.sh`) + +Draait op udev REMOVE-trigger. Checkt of `/mnt/sdcard` een stale mount is (device-node +weg of gemount op een verwijderd device) en ruimt hem dan op: service stoppen, +`umount` (fallback `umount -l`). Bij een gezonde mount (ander device) doet het niets. + ### Systemd service (`/etc/systemd/system/dashcam-transfer.service`) ```ini @@ -121,9 +165,20 @@ Alle activiteit wordt gelogd naar `/tmp/dashcam-transfer.log` op nextcloud. ## Ontwikkeling Lokale repo: `~/src/dashcam-transfer/` op Mac Mini. +Tests: `python3 -m unittest discover -s tests`, `./tests/test_mount_script.sh` +en `./tests/test_cleanup_script.sh`. Deployen naar nextcloud: ```bash scp dashcam-transfer.py aiaiaiaiagent@nextcloud:/tmp/ ssh aiaiaiaiagent@nextcloud 'sudo -A cp /tmp/dashcam-transfer.py /usr/local/bin/dashcam-transfer' +scp dashcam-transfer-mount.sh aiaiaiaiagent@nextcloud:/tmp/ +ssh aiaiaiaiagent@nextcloud 'sudo -A cp /tmp/dashcam-transfer-mount.sh /usr/local/bin/dashcam-transfer-mount' +scp dashcam-transfer-cleanup.sh aiaiaiaiagent@nextcloud:/tmp/ +ssh aiaiaiaiagent@nextcloud 'sudo -A cp /tmp/dashcam-transfer-cleanup.sh /usr/local/bin/dashcam-transfer-cleanup' +scp udev/99-dashcam-trigger.rules aiaiaiaiagent@nextcloud:/tmp/ +ssh aiaiaiaiagent@nextcloud 'sudo -A cp /tmp/99-dashcam-trigger.rules /etc/udev/rules.d/ && sudo -A udevadm control --reload-rules' ``` + +Controleer na deploy altijd met `sudo sha256sum` dat de hash op de server gelijk +is aan de lokale `shasum -a 256`. diff --git a/dashcam-transfer-cleanup.sh b/dashcam-transfer-cleanup.sh new file mode 100755 index 0000000..436f2c9 --- /dev/null +++ b/dashcam-transfer-cleanup.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# Remove-trigger cleanup for the dashcam SD card. +# Called by udev rule 99-dashcam-trigger.rules via systemd-run on +# ACTION=="remove" with %k as $1. Unmounts a stale /mnt/sdcard mount and +# stops a transfer that can no longer complete. +# Idempotent: does nothing when /mnt/sdcard is not mounted or still healthy. +logger -t dashcam-transfer "Remove trigger: block device weggevallen" + +src=$(findmnt -no SOURCE /mnt/sdcard 2>/dev/null) +if [ -z "$src" ]; then + logger -t dashcam-transfer "Remove: /mnt/sdcard niet gemount, niets te doen" + exit 0 +fi + +case "$src" in + /*) ;; + *) logger -t dashcam-transfer "Remove: onbekende source ($src), niets te doen"; exit 0 ;; +esac + +stale=0 +if [ ! -e "$src" ]; then + stale=1 +elif [ -n "$1" ] && [[ "$src" == *"/$1"* ]]; then + stale=1 +fi + +if [ "$stale" -eq 0 ]; then + logger -t dashcam-transfer "Remove: device $src nog aanwezig, mount blijft" + exit 0 +fi + +logger -t dashcam-transfer "Remove: device $src verdwenen — transfer stoppen, mount opruimen" +systemctl stop dashcam-transfer.service 2>/dev/null || true +sleep 1 +if ! umount /mnt/sdcard 2>/dev/null; then + logger -t dashcam-transfer "Remove: umount mislukt, lazy unmount" + umount -l /mnt/sdcard 2>/dev/null || true +fi +logger -t dashcam-transfer "Remove: stale mount opgeruimd" diff --git a/dashcam-transfer-mount.sh b/dashcam-transfer-mount.sh index 559a742..c4126f0 100755 --- a/dashcam-transfer-mount.sh +++ b/dashcam-transfer-mount.sh @@ -1,15 +1,22 @@ #!/bin/bash -# Mount SD card and start dashcam transfer. +# Mount SD card by kernel device name (udev %k) and start dashcam transfer. # Called by udev rule 99-dashcam-trigger.rules via systemd-run. # systemd-run is used (not direct udev exec) because udev context # lacks mount/umount permissions on Fedora with SELinux enforcing. -# -# The udev rule passes the kernel name (e.g. "sde1") as $1 so the -# device is mounted dynamically, independent of its filesystem UUID. +MAX_WAIT="${DASHCAM_MAX_WAIT:-60}" + DEV=${1:+/dev/$1} logger -t dashcam-transfer "Trigger: SD card detected ($DEV)" + +# Never interrupt a running transfer: a flaky reader can cause a second +# udev event (USB re-enumeration) which must not unmount a busy card. +if systemctl is-active --quiet dashcam-transfer.service; then + logger -t dashcam-transfer "Trigger: transfer service al bezig — skip" + exit 0 +fi + if [ -n "$1" ]; then - for ((i=0; i<60; i++)); do + for ((i=0; i/dev/null) DEV=/dev/disk/by-uuid/$UUID - for ((i=0; i<60; i++)); do + for ((i=0; i/dev/null; do umount /mnt/sdcard 2>/dev/null || break done + mount "$DEV" /mnt/sdcard -o uid=1000,gid=1000,fmask=0000,dmask=0000 if [ $? -eq 0 ]; then logger -t dashcam-transfer "Trigger: mounted, starting transfer" + # Read-only exFAT health check; logs warnings but never blocks the transfer. + if command -v fsck.exfat >/dev/null 2>&1; then + fsck.exfat -n "$DEV" 2>&1 | logger -t dashcam-transfer + logger -t dashcam-transfer "Trigger: fsck.exfat exit ${PIPESTATUS[0]}" + fi systemctl start dashcam-transfer else logger -t dashcam-transfer "TRIGGER ERROR: mount failed" -fi \ No newline at end of file +fi diff --git a/dashcam-transfer.py b/dashcam-transfer.py index 8279d80..ac49b61 100644 --- a/dashcam-transfer.py +++ b/dashcam-transfer.py @@ -8,8 +8,8 @@ import logging import os import re import shutil -import subprocess import sys +import time import urllib.request SD_CARD = "/mnt/sdcard" @@ -22,6 +22,27 @@ LOCK_FILE = "/tmp/dashcam-transfer.lock" CAMERA_MAP = {"F": "front", "B": "back"} FILE_PATTERN = re.compile(r"^(\d{8})\d{6}_\d+([FB])\.MP4$") +MAX_RETRIES = 3 +RETRY_DELAY = 2 +MAX_CONSECUTIVE_ERRORS = 5 + +WHATSAPP_API = "http://localhost:3001/client/sendMessage/EdGpt" # localhost = nextcloud server, niet Mac Mini! +WHATSAPP_CHAT = "31620248011@c.us" + +UNREADABLE_REASON = ( + "SD-kaart onleesbaar (I/O-fout): lezer/kaart defect of niet netjes afgemeld. " + "fsck.exfat en lezer/poort controleren. Geen bronbestanden verwijderd." +) + + +class TransferAborted(Exception): + def __init__(self, ok, errors, consecutive, reason): + self.ok = ok + self.errors = errors + self.consecutive = consecutive + self.reason = reason + super().__init__(reason) + def setup_logging(): logging.basicConfig( @@ -49,16 +70,37 @@ def check_mount(path, name): def get_mp4_files(src_dir): + """Return matching (fname, date, camera) tuples, [] when absent, None on I/O error.""" if not os.path.isdir(src_dir): return [] + try: + entries = os.listdir(src_dir) + except OSError as e: + logging.error("SD-kaart directory niet leesbaar: %s", e) + return None result = [] - for fname in os.listdir(src_dir): + for fname in entries: match = FILE_PATTERN.match(fname) if match: result.append((fname, match.group(1), match.group(2).upper())) return result +def check_source_readable(src_dir, files): + """Smoke-test that the SD card actually returns data, before freeing space.""" + try: + with os.scandir(src_dir): + pass + if files: + sample = os.path.join(src_dir, files[0][0]) + with open(sample, "rb") as f: + f.read(1) + except OSError as e: + logging.error("SD-kaart niet leesbaar (I/O-fout): %s", e) + return False + return True + + def ensure_free_space(needed_bytes, target_base): stat = os.statvfs(target_base) free = stat.f_frsize * stat.f_bavail @@ -105,19 +147,113 @@ def ensure_free_space(needed_bytes, target_base): logging.info("Ruimte vrijgemaakt: %d bytes door %d mappen", freed, len(removed)) -WHATSAPP_API = "http://localhost:3001/client/sendMessage/EdGpt" # localhost = nextcloud server, niet Mac Mini! -WHATSAPP_CHAT = "31620248011@c.us" +def copy_file(src, dst, attempts=MAX_RETRIES, delay=RETRY_DELAY): + """Copy src → dst with retry. Raises the last OSError when all attempts fail.""" + last_error = None + for attempt in range(1, attempts + 1): + try: + shutil.copy2(src, dst) + return + except OSError as e: + last_error = e + logging.warning( + "Kopieerpoging %d/%d mislukt voor %s: %s", + attempt, attempts, os.path.basename(src), e, + ) + if attempt < attempts: + time.sleep(delay) + raise last_error -def send_whatsapp_notification(ok, errors, total_bytes): - if errors == 0: - prefix = "✅" - status = "geen fouten" - else: - prefix = "❌" - status = f"{errors} fout(en)" +def _safe_remove(path, name): + try: + os.remove(path) + except OSError as e: + logging.warning("Bron %s kon niet verwijderd worden (blijft op kaart): %s", name, e) + + +def transfer_files(files, source_dir, target_base): + """Copy+verify+delete per file. Returns (ok, errors) or raises TransferAborted.""" + ok = 0 + errors = 0 + consecutive = 0 + last_error = None + + for fname, date_str, cam in files: + cam_name = CAMERA_MAP[cam] + target_dir = os.path.join(target_base, date_str, cam_name) + src = os.path.join(source_dir, fname) + dst = os.path.join(target_dir, fname) + + try: + os.makedirs(target_dir, exist_ok=True) + + if os.path.exists(dst): + src_size = os.path.getsize(src) + dst_size = os.path.getsize(dst) + if src_size == dst_size: + logging.info( + "Bestaat al en klopt: %s → %s/%s/", fname, date_str, cam_name + ) + _safe_remove(src, fname) + ok += 1 + consecutive = 0 + continue + logging.warning( + "Doel %s bestaat maar is anders (%d vs %d bytes) — overschrijf", + dst, dst_size, src_size, + ) + os.remove(dst) + + copy_file(src, dst) + src_size = os.path.getsize(src) + dst_size = os.path.getsize(dst) + if dst_size != src_size: + raise OSError("Verificatie mislukt: %d vs %d bytes" % (src_size, dst_size)) + except OSError as e: + errors += 1 + consecutive += 1 + last_error = e + logging.error("Fout bij overzetten %s: %s", fname, e) + if consecutive >= MAX_CONSECUTIVE_ERRORS: + logging.error( + "Te veel opeenvolgende fouten (%d) — transfer gestopt", consecutive + ) + raise TransferAborted( + ok, errors, consecutive, + f"SD-kaart onleesbaar (I/O-fout) — gestopt na {consecutive} " + f"opeenvolgende fouten. Laatste fout: {last_error}. " + f"Geen bronbestanden verwijderd.", + ) + continue + + _safe_remove(src, fname) + logging.info( + "Overgezet: %s → %s/%s/ (%d bytes)", fname, date_str, cam_name, src_size + ) + ok += 1 + consecutive = 0 + + return ok, errors + + +def send_whatsapp_notification(ok, errors, total_bytes, aborted=False, reason=None): gb = total_bytes / (1024 ** 3) - body = f"{prefix} Dashcam transfer voltooid: {ok} bestanden ({gb:.1f} GB) overgezet, {status}" + if aborted: + body = ( + f"❌ Dashcam transfer GESTOPT: {reason} " + f"({ok} ok, {errors} fout(en) tot dusver)" + ) + elif errors == 0: + body = ( + f"✅ Dashcam transfer voltooid: {ok} bestanden ({gb:.1f} GB) " + f"overgezet, geen fouten" + ) + else: + body = ( + f"❌ Dashcam transfer voltooid: {ok} bestanden ({gb:.1f} GB) " + f"overgezet, {errors} fout(en)" + ) payload = {"chatId": WHATSAPP_CHAT, "content": body, "contentType": "string"} try: data = json.dumps(payload).encode() @@ -148,6 +284,9 @@ def main(): check_mount(SSD, "Ultratouch SSD") files = get_mp4_files(SOURCE_DIR) + if files is None: + send_whatsapp_notification(0, 0, 0, aborted=True, reason=UNREADABLE_REASON) + return if not files: logging.info("Geen MP4 bestanden gevonden in %s", SOURCE_DIR) return @@ -155,63 +294,36 @@ def main(): if args.limit > 0: files = files[:args.limit] - total_size = sum( - os.path.getsize(os.path.join(SOURCE_DIR, fname)) for fname, _, _ in files - ) - logging.info( - "Start transfer: %d bestanden, %d bytes", len(files), total_size - ) - - ensure_free_space(total_size, TARGET_BASE) - - ok = 0 - errors = 0 - for fname, date_str, cam in files: - target_dir = os.path.join(TARGET_BASE, date_str, CAMERA_MAP[cam]) - os.makedirs(target_dir, exist_ok=True) - src = os.path.join(SOURCE_DIR, fname) - dst = os.path.join(target_dir, fname) - - if os.path.exists(dst): - src_size = os.path.getsize(src) - dst_size = os.path.getsize(dst) - if src_size == dst_size: - logging.info( - "Bestaat al en klopt: %s → %s/%s/", fname, date_str, CAMERA_MAP[cam] - ) - os.remove(src) - ok += 1 - continue - else: - logging.warning( - "Doel %s bestaat maar is anders (%d vs %d bytes) — overschrijf", - dst, dst_size, src_size, - ) - os.remove(dst) + if not check_source_readable(SOURCE_DIR, files): + send_whatsapp_notification(0, 0, 0, aborted=True, reason=UNREADABLE_REASON) + return + total_size = 0 + for fname, _, _ in files: try: - subprocess.run( - ["cp", src, dst], check=True, capture_output=True, timeout=600 - ) - except (subprocess.CalledProcessError, OSError) as e: - logging.error("Fout bij kopiëren %s: %s", fname, e) - errors += 1 - continue + total_size += os.path.getsize(os.path.join(SOURCE_DIR, fname)) + except OSError as e: + logging.error("Grootte van %s niet leesbaar: %s", fname, e) + send_whatsapp_notification(0, 0, 0, aborted=True, reason=UNREADABLE_REASON) + return - dst_size = os.path.getsize(dst) - src_size = os.path.getsize(src) - if dst_size != src_size: - logging.error( - "Verificatie mislukt %s: %d vs %d bytes", fname, src_size, dst_size - ) - errors += 1 - continue + logging.info("Start transfer: %d bestanden, %d bytes", len(files), total_size) - os.remove(src) - logging.info( - "Overgezet: %s → %s/%s/ (%d bytes)", fname, date_str, CAMERA_MAP[cam], src_size + try: + ensure_free_space(total_size, TARGET_BASE) + except SystemExit: + send_whatsapp_notification( + 0, 0, total_size, aborted=True, + reason="Onvoldoende vrije ruimte op de SSD, ook na opschonen.", ) - ok += 1 + raise + + try: + ok, errors = transfer_files(files, SOURCE_DIR, TARGET_BASE) + except TransferAborted as e: + logging.info("Transfer gestopt: %d ok, %d fouten", e.ok, e.errors) + send_whatsapp_notification(e.ok, e.errors, total_size, aborted=True, reason=e.reason) + return logging.info("Transfer voltooid: %d ok, %d fouten", ok, errors) send_whatsapp_notification(ok, errors, total_size) diff --git a/tests/test_cleanup_script.sh b/tests/test_cleanup_script.sh new file mode 100755 index 0000000..914396e --- /dev/null +++ b/tests/test_cleanup_script.sh @@ -0,0 +1,118 @@ +#!/bin/bash +# Functional test for dashcam-transfer-cleanup.sh using fake tools on PATH. +set -u + +SCRIPT="$(cd "$(dirname "$0")/.." && pwd)/dashcam-transfer-cleanup.sh" +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +FAKEBIN="$WORK/bin" +mkdir -p "$FAKEBIN" + +LOGGER_FILE="$WORK/logger.log" +UMOUNT_MARKER="$WORK/umounted" +LAZY_MARKER="$WORK/lazy" +STOP_MARKER="$WORK/stopped" + +export LOGGER_FILE UMOUNT_MARKER LAZY_MARKER STOP_MARKER FAKE_SRC FAKE_UMOUNT_FAIL + +cat > "$FAKEBIN/logger" <<'EOF' +#!/bin/bash +echo "logger: $*" >> "$LOGGER_FILE" +EOF + +cat > "$FAKEBIN/findmnt" <<'EOF' +#!/bin/bash +if [ -z "$FAKE_SRC" ]; then + exit 1 +fi +echo "$FAKE_SRC" +EOF + +cat > "$FAKEBIN/systemctl" <<'EOF' +#!/bin/bash +if [ "$1" = "stop" ]; then + touch "$STOP_MARKER" +fi +EOF + +cat > "$FAKEBIN/umount" <<'EOF' +#!/bin/bash +if [ "${1:-}" = "-l" ]; then + touch "$LAZY_MARKER" + exit 0 +fi +if [ "$FAKE_UMOUNT_FAIL" = "1" ]; then + exit 1 +fi +touch "$UMOUNT_MARKER" +EOF + +chmod +x "$FAKEBIN"/* + +pass=0 +fail=0 +check() { # $1=desc, $2=condition + if eval "$2"; then + echo "PASS: $1" + pass=$((pass + 1)) + else + echo "FAIL: $1" + fail=$((fail + 1)) + fi +} + +reset() { + rm -f "$LOGGER_FILE" "$UMOUNT_MARKER" "$LAZY_MARKER" "$STOP_MARKER" + unset FAKE_UMOUNT_FAIL +} + +# Test 1: niet gemount → niets te doen +reset +FAKE_SRC="" +PATH="$FAKEBIN:$PATH" "$SCRIPT" +check "niet gemount → geen umount" 'test ! -e "$UMOUNT_MARKER"' +check "niet gemount → geen service stop" 'test ! -e "$STOP_MARKER"' + +# Test 2: device nog aanwezig → mount blijft +reset +FAKE_SRC="$WORK/fake-device" +touch "$FAKE_SRC" +PATH="$FAKEBIN:$PATH" "$SCRIPT" +check "device aanwezig → geen umount" 'test ! -e "$UMOUNT_MARKER"' +check "device aanwezig → geen service stop" 'test ! -e "$STOP_MARKER"' + +# Test 3: device verdwenen → stop + umount +reset +FAKE_SRC="$WORK/gone-device" +PATH="$FAKEBIN:$PATH" "$SCRIPT" +check "device weg → service gestopt" 'test -e "$STOP_MARKER"' +check "device weg → umount uitgevoerd" 'test -e "$UMOUNT_MARKER"' +check "device weg → geen lazy nodig" 'test ! -e "$LAZY_MARKER"' + +# Test 4: umount mislukt → lazy umount +reset +FAKE_SRC="$WORK/gone-device" +export FAKE_UMOUNT_FAIL=1 +PATH="$FAKEBIN:$PATH" "$SCRIPT" +check "umount faalt → lazy umount" 'test -e "$LAZY_MARKER"' + +# Test 5: node bestaat nog maar source = verwijderd device → stale +reset +mkdir -p "$WORK/dev" +touch "$WORK/dev/sde1" +FAKE_SRC="$WORK/dev/sde1" +PATH="$FAKEBIN:$PATH" "$SCRIPT" sde1 +check "source matcht verwijderd device → umount" 'test -e "$UMOUNT_MARKER"' +check "source matcht verwijderd device → service gestopt" 'test -e "$STOP_MARKER"' + +# Test 6: ander device verwijdert, sdcard-healthy → niets doen +reset +touch "$WORK/dev/sdb2" +FAKE_SRC="$WORK/dev/sde1" +PATH="$FAKEBIN:$PATH" "$SCRIPT" sdb2 +check "ander device → geen umount" 'test ! -e "$UMOUNT_MARKER"' +check "ander device → geen service stop" 'test ! -e "$STOP_MARKER"' + +echo "=== $pass pass, $fail fail ===" +[ "$fail" -eq 0 ] diff --git a/tests/test_dashcam_transfer.py b/tests/test_dashcam_transfer.py new file mode 100644 index 0000000..bcf6f2f --- /dev/null +++ b/tests/test_dashcam_transfer.py @@ -0,0 +1,251 @@ +import errno +import importlib.util +import json +import logging +import os +import sys +import tempfile +import unittest +from unittest import mock + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +MODULE_PATH = os.path.join(REPO_ROOT, "dashcam-transfer.py") + +spec = importlib.util.spec_from_file_location("dashcam_transfer", MODULE_PATH) +dashcam = importlib.util.module_from_spec(spec) +spec.loader.exec_module(dashcam) +sys.modules["dashcam_transfer"] = dashcam + +logging.getLogger().addHandler(logging.NullHandler()) + + +def make_mp4(path, size): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "wb") as f: + f.write(b"0" * size) + + +class GetMp4FilesTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + + def tearDown(self): + self.tmp.cleanup() + + def test_matches_front_and_back_files(self): + src = self.tmp.name + make_mp4(os.path.join(src, "20260622050346_0000001F.MP4"), 10) + make_mp4(os.path.join(src, "20260622050346_0000002B.MP4"), 10) + make_mp4(os.path.join(src, "negeer.txt"), 10) + result = dashcam.get_mp4_files(src) + self.assertEqual(len(result), 2) + self.assertIn(("20260622050346_0000001F.MP4", "20260622", "F"), result) + self.assertIn(("20260622050346_0000002B.MP4", "20260622", "B"), result) + + def test_missing_dir_returns_empty_list(self): + self.assertEqual(dashcam.get_mp4_files(os.path.join(self.tmp.name, "nope")), []) + + def test_listdir_error_returns_none(self): + with mock.patch.object(dashcam.os, "listdir", side_effect=OSError(errno.EIO, "I/O error")): + self.assertIsNone(dashcam.get_mp4_files(self.tmp.name)) + + +class CheckSourceReadableTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.src = self.tmp.name + + def tearDown(self): + self.tmp.cleanup() + + def test_readable_dir_returns_true(self): + fname = "20260622050346_0000001F.MP4" + make_mp4(os.path.join(self.src, fname), 10) + self.assertTrue(dashcam.check_source_readable(self.src, [(fname, "20260622", "F")])) + + def test_io_error_on_read_returns_false(self): + fname = "20260622050346_0000001F.MP4" + make_mp4(os.path.join(self.src, fname), 10) + with mock.patch("builtins.open", side_effect=OSError(errno.EIO, "I/O error")): + self.assertFalse(dashcam.check_source_readable(self.src, [(fname, "20260622", "F")])) + + +class CopyFileTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.src = os.path.join(self.tmp.name, "src.mp4") + self.dst = os.path.join(self.tmp.name, "dst.mp4") + make_mp4(self.src, 100) + + def tearDown(self): + self.tmp.cleanup() + + def test_copies_file(self): + dashcam.copy_file(self.src, self.dst, attempts=2, delay=0) + self.assertEqual(os.path.getsize(self.dst), 100) + + def test_retries_then_succeeds(self): + real_copy2 = dashcam.shutil.copy2 + calls = {"n": 0} + + def flaky(s, d): + calls["n"] += 1 + if calls["n"] < 3: + raise OSError(errno.EIO, "I/O error") + real_copy2(s, d) + + with mock.patch.object(dashcam.shutil, "copy2", side_effect=flaky): + dashcam.copy_file(self.src, self.dst, attempts=3, delay=0) + self.assertEqual(calls["n"], 3) + self.assertEqual(os.path.getsize(self.dst), 100) + + def test_raises_after_exhausting_attempts(self): + with mock.patch.object(dashcam.shutil, "copy2", side_effect=OSError(errno.EIO, "I/O error")): + with self.assertRaises(OSError): + dashcam.copy_file(self.src, self.dst, attempts=2, delay=0) + + +class TransferFilesTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.src = os.path.join(self.tmp.name, "src") + self.base = os.path.join(self.tmp.name, "base") + os.makedirs(self.src) + os.makedirs(self.base) + + def tearDown(self): + self.tmp.cleanup() + + def _make(self, fname, size): + make_mp4(os.path.join(self.src, fname), size) + return (fname, fname[:8], fname[-5]) + + def test_transfers_files_to_camera_subdirs(self): + files = [ + self._make("20260622050346_0000001F.MP4", 100), + self._make("20260622050346_0000002B.MP4", 200), + ] + ok, errors = dashcam.transfer_files(files, self.src, self.base) + self.assertEqual((ok, errors), (2, 0)) + self.assertFalse(os.path.exists(os.path.join(self.src, files[0][0]))) + self.assertEqual( + os.path.getsize(os.path.join(self.base, "20260622", "front", files[0][0])), 100 + ) + self.assertEqual( + os.path.getsize(os.path.join(self.base, "20260622", "back", files[1][0])), 200 + ) + + def test_skips_existing_matching_destination(self): + files = [self._make("20260622050346_0000001F.MP4", 100)] + dst = os.path.join(self.base, "20260622", "front", files[0][0]) + make_mp4(dst, 100) + ok, errors = dashcam.transfer_files(files, self.src, self.base) + self.assertEqual((ok, errors), (1, 0)) + self.assertFalse(os.path.exists(os.path.join(self.src, files[0][0]))) + self.assertEqual(os.path.getsize(dst), 100) + + def test_overwrites_existing_different_destination(self): + files = [self._make("20260622050346_0000001F.MP4", 100)] + dst = os.path.join(self.base, "20260622", "front", files[0][0]) + make_mp4(dst, 10) + ok, errors = dashcam.transfer_files(files, self.src, self.base) + self.assertEqual((ok, errors), (1, 0)) + self.assertEqual(os.path.getsize(dst), 100) + + def test_verification_mismatch_counts_as_error_and_keeps_source(self): + files = [self._make("20260622050346_0000001F.MP4", 100)] + + def bad_copy2(s, d): + with open(d, "wb") as f: + f.write(b"0" * 50) + + with mock.patch.object(dashcam.shutil, "copy2", side_effect=bad_copy2), mock.patch.object( + dashcam.time, "sleep" + ): + ok, errors = dashcam.transfer_files(files, self.src, self.base) + self.assertEqual((ok, errors), (0, 1)) + self.assertTrue(os.path.exists(os.path.join(self.src, files[0][0]))) + + def test_aborts_after_max_consecutive_errors(self): + files = [ + self._make("2026062205034%d_000000%dF.MP4" % (i, i + 1), 10) + for i in range(dashcam.MAX_CONSECUTIVE_ERRORS + 3) + ] + with mock.patch.object( + dashcam.shutil, "copy2", side_effect=OSError(errno.EIO, "I/O error") + ), mock.patch.object(dashcam.time, "sleep"): + with self.assertRaises(dashcam.TransferAborted) as ctx: + dashcam.transfer_files(files, self.src, self.base) + self.assertEqual(ctx.exception.consecutive, dashcam.MAX_CONSECUTIVE_ERRORS) + self.assertEqual(ctx.exception.ok, 0) + for fname, _, _ in files: + self.assertTrue(os.path.exists(os.path.join(self.src, fname))) + + +class EnsureFreeSpaceTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.base = self.tmp.name + + def tearDown(self): + self.tmp.cleanup() + + @staticmethod + def _statvfs(free_bytes): + frsize = 4096 + return mock.Mock(f_frsize=frsize, f_bavail=free_bytes // frsize) + + def test_no_deletion_when_enough_space(self): + with mock.patch.object(dashcam.os, "statvfs", return_value=self._statvfs(10_000_000)): + dashcam.ensure_free_space(1000, self.base) + + def test_deletes_oldest_date_dirs_until_enough(self): + for name, fname in (("20260301", "a.mp4"), ("20260310", "b.mp4"), ("20260811", "c.mp4")): + make_mp4(os.path.join(self.base, name, fname), 5000) + with mock.patch.object(dashcam.os, "statvfs", return_value=self._statvfs(4096)): + dashcam.ensure_free_space(5000, self.base) + self.assertFalse(os.path.exists(os.path.join(self.base, "20260301"))) + self.assertTrue(os.path.exists(os.path.join(self.base, "20260310"))) + self.assertTrue(os.path.exists(os.path.join(self.base, "20260811"))) + + def test_exits_when_still_insufficient(self): + make_mp4(os.path.join(self.base, "20260301", "a.mp4"), 100) + with mock.patch.object(dashcam.os, "statvfs", return_value=self._statvfs(0)): + with self.assertRaises(SystemExit): + dashcam.ensure_free_space(10_000, self.base) + + +class SendWhatsappTest(unittest.TestCase): + def setUp(self): + patcher = mock.patch.object(dashcam.urllib.request, "urlopen") + self.mock_urlopen = patcher.start() + self.mock_urlopen.return_value.__enter__.return_value = mock.Mock() + self.addCleanup(patcher.stop) + + def _body(self): + req = self.mock_urlopen.call_args[0][0] + return json.loads(req.data)["content"] + + def test_success_message(self): + dashcam.send_whatsapp_notification(5, 0, 381_000_000) + self.assertIn("5 bestanden", self._body()) + self.assertIn("geen fouten", self._body()) + + def test_errors_message(self): + dashcam.send_whatsapp_notification(3, 2, 100_000_000) + self.assertIn("2 fout(en)", self._body()) + + def test_aborted_message(self): + dashcam.send_whatsapp_notification( + 1, 4, 100_000_000, aborted=True, reason="SD-kaart onleesbaar" + ) + self.assertIn("GESTOPT", self._body()) + self.assertIn("SD-kaart onleesbaar", self._body()) + + def test_api_failure_is_swallowed(self): + self.mock_urlopen.side_effect = Exception("boom") + dashcam.send_whatsapp_notification(0, 0, 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mount_script.sh b/tests/test_mount_script.sh new file mode 100755 index 0000000..01d97ac --- /dev/null +++ b/tests/test_mount_script.sh @@ -0,0 +1,98 @@ +#!/bin/bash +# Functional test for dashcam-transfer-mount.sh using fake tools on PATH. +set -u + +SCRIPT="$(cd "$(dirname "$0")/.." && pwd)/dashcam-transfer-mount.sh" +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +FAKEBIN="$WORK/bin" +mkdir -p "$FAKEBIN" + +LOGGER_FILE="$WORK/logger.log" +MOUNT_LOG="$WORK/mount.log" +FSCK_LOG="$WORK/fsck.log" +MOUNT_MARKER="$WORK/mounted" +START_MARKER="$WORK/started" + +export LOGGER_FILE MOUNT_LOG FSCK_LOG MOUNT_MARKER START_MARKER + +cat > "$FAKEBIN/logger" <<'EOF' +#!/bin/bash +echo "logger: $*" >> "$LOGGER_FILE" +EOF + +cat > "$FAKEBIN/systemctl" <<'EOF' +#!/bin/bash +if [ "$1" = "is-active" ]; then + case "$2" in + --quiet) ;; + *) echo "$SYSTEMCTL_STATE" ;; + esac + [ "$SYSTEMCTL_STATE" = "active" ] +elif [ "$1" = "start" ]; then + touch "$START_MARKER" +fi +EOF + +cat > "$FAKEBIN/mount" <<'EOF' +#!/bin/bash +echo "mount $*" >> "$MOUNT_LOG" +touch "$MOUNT_MARKER" +exit 0 +EOF + +cat > "$FAKEBIN/umount" <<'EOF' +#!/bin/bash +exit 0 +EOF + +cat > "$FAKEBIN/mountpoint" <<'EOF' +#!/bin/bash +exit 1 +EOF + +cat > "$FAKEBIN/fsck.exfat" <<'EOF' +#!/bin/bash +echo "fsck $*" >> "$FSCK_LOG" +exit 0 +EOF + +cat > "$FAKEBIN/lsblk" <<'EOF' +#!/bin/bash +exit 0 +EOF + +chmod +x "$FAKEBIN"/* + +pass=0 +fail=0 +check() { # $1=desc, $2=condition + if eval "$2"; then + echo "PASS: $1" + pass=$((pass + 1)) + else + echo "FAIL: $1" + fail=$((fail + 1)) + fi +} + +# Test 1: service already active → skip, never unmount/mount/start +export SYSTEMCTL_STATE=active +rm -f "$MOUNT_MARKER" "$START_MARKER" "$LOGGER_FILE" "$MOUNT_LOG" "$FSCK_LOG" +PATH="$FAKEBIN:$PATH" DASHCAM_MAX_WAIT=0 "$SCRIPT" sde1 +check "service actief → exit zonder mount" 'test ! -e "$MOUNT_MARKER"' +check "service actief → geen service start" 'test ! -e "$START_MARKER"' +check "service actief → skip gelogd" 'grep -q "al bezig" "$LOGGER_FILE"' + +# Test 2: service inactive → mount, fsck, start +export SYSTEMCTL_STATE=inactive +rm -f "$MOUNT_MARKER" "$START_MARKER" "$LOGGER_FILE" "$MOUNT_LOG" "$FSCK_LOG" +PATH="$FAKEBIN:$PATH" DASHCAM_MAX_WAIT=0 "$SCRIPT" sde1 +check "service inactief → gemount" 'test -e "$MOUNT_MARKER"' +check "service inactief → service gestart" 'test -e "$START_MARKER"' +check "fsck.exfat uitgevoerd" 'test -s "$FSCK_LOG"' +check "mount gelogd" 'grep -q "mounted, starting transfer" "$LOGGER_FILE"' + +echo "=== $pass pass, $fail fail ===" +[ "$fail" -eq 0 ] diff --git a/udev/99-dashcam-trigger.rules b/udev/99-dashcam-trigger.rules new file mode 100644 index 0000000..29ad4c3 --- /dev/null +++ b/udev/99-dashcam-trigger.rules @@ -0,0 +1,12 @@ +# Dashcam SD card trigger — deploy naar /etc/udev/rules.d/99-dashcam-trigger.rules +# udev draait deze via systemd-run (niet direct) omdat de udev-context geen +# mount/umount-rechten heeft op Fedora met SELinux enforcing. +# %k = kernel device name (bv. sde1), doorgegeven als $1. + +# ADD: kaart ingeplugd → mount + start transfer +ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="14cd", ENV{ID_MODEL_ID}=="1212", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-mount %k" +ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="05e3", ENV{ID_MODEL_ID}=="0751", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-mount %k" + +# REMOVE: kaartlezer eruit → stale mount opruimen + mislukte transfer stoppen +ACTION=="remove", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="14cd", ENV{ID_MODEL_ID}=="1212", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-cleanup %k" +ACTION=="remove", SUBSYSTEM=="block", KERNEL=="sd*[0-9]", ENV{ID_VENDOR_ID}=="05e3", ENV{ID_MODEL_ID}=="0751", RUN+="/usr/bin/systemd-run --no-block --collect /usr/local/bin/dashcam-transfer-cleanup %k"