Skip to content

Always finish a Borg job and unblock the UI when it raisesΒ #2600

Closed

Description

@m3nu

Line numbers are as of commit 9c4fc1e.

Problem

If a job raises inside job.run(), nothing reports that the job finished.

  • SiteWorker.run (src/vorta/borg/jobs_manager.py:52-65) only catches queue.Empty. Any other exception leaves the loop, and the threading.Thread dies.
  • Only sys.excepthook is installed (src/vorta/__main__.py:38). threading.excepthook is not set anywhere, so the traceback goes to stderr only. It does not reach the log file, and no ExceptionDialog appears. In background mode or from the app bundle, that means it is lost.
  • BorgJob._run (src/vorta/borg/borg_job.py:262-386) has no try/finally. By the time Popen runs (:282), started_event() (:263) has already emitted backup_started_event for most job types, and an EventLogModel row has been saved (:265-270). If anything raises after that:
    • result and app.backup_finished_event are never emitted (:384), so MainWindow.backup_finished_event (src/vorta/views/main_window.py:339-344) never re-enables the Start Backup and archive buttons.
    • The tray icon stays in its active state (src/vorta/application.py:156-161).
    • The event log row keeps no returncode and no end_time (:375-381).
    • cleanup_files are never closed (:385-386). For extract, these hold the temporary pattern file.
    • For a scheduled backup, SchedulerExecution.notify never runs (src/vorta/scheduler/execution.py:99). The timer was already removed before the job was queued (:96), so that profile is not re-armed by this run. The JobModel row stays running until the next start, when src/vorta/store/connection.py:62-64 marks it interrupted.
    • For post-backup tasks, _post_backup_pending never reaches zero (execution.py:224-246), so the "Post Backup Tasks ..." notification never appears.
  • Jobs still queued for that repo are not lost. is_worker_running() now returns False because the thread is dead (jobs_manager.py:97), and the next add_job for the same repo starts a new SiteWorker on the same queue (:122-126). Until another job is added for that repo, though, the queued jobs do not run.

This has happened in practice. #1766 shows Exception in thread Thread-1 from a KeyError in the stderr parsing loop of BorgVersionJob. That particular key was fixed later, but the failure mode is still there.

Proposed change

  1. In BorgJob.run/_run, finalise every job even when it raises:
    • Wrap everything after started_event() in try/except/finally.
    • On an exception, call logger.exception(...) so the traceback reaches the log file.
    • Build a result with the same keys as a normal one: params, cmd, returncode set to a non-success value (for example 2), errors=[(logging.CRITICAL, '<ExceptionType>: <message>')], data set to {} or '', and log_entry_id.
    • Save the event log row with that returncode and end_time.
    • Emit the result only once. Keep a flag that is set when finished_event has been entered. If the exception came from finished_event itself, fall back to emitting self.result, plus app.backup_finished_event when this job type emits it normally, without calling finished_event again.
    • Close cleanup_files in finally.
  2. In SiteWorker.run, catch Exception around job.run(), log it, clear current_job, and move on to the next queued job. This is the backstop for any JobInterface that is not a BorgJob. Today BorgJob is the only implementation in src/.
  3. Keep bugs visible. Route the exception to the main thread through a queued Qt signal, so the existing ExceptionDialog and file log still appear (a worker thread must not create the dialog itself). Also set threading.excepthook to log through logger.critical as a last resort.

Bugs this fixes

Each of these currently leaves the UI blocked as described above. After the change, each one ends as a failed job with a logged traceback. The root causes still need their own fixes.

  • Extract to a folder that no longer exists. BorgExtractJob.prepare passes cwd=destination_folder (src/vorta/borg/extract.py:93). If that folder is removed or unmounted before the queued job starts, Popen (borg_job.py:282-291) raises FileNotFoundError after backup_started_event and after the event log row has been saved. The Borg binary disappearing between prepare() and run() (cmd[0] is set at borg_job.py:102) has the same effect.
  • A stderr line that is valid JSON but not a dict with type. parsed['type'] (borg_job.py:316-318) raises TypeError for something like 0 or "x", and KeyError for a dict without type. Only JSONDecodeError is caught (:351). This is unlikely with Borg's own --log-json output, but stderr also carries output from SSH and remote wrappers.
  • Row removed while the job was queued.
    • BorgPruneJob.finished_event and BorgDeleteJob.finished_event call RepoModel.get(id=...) before they emit backup_finished_event (src/vorta/borg/prune.py:14, src/vorta/borg/delete.py:16).
    • BorgRenameJob.process_result calls RepoModel.get and ArchiveModel.get (src/vorta/borg/rename.py:43-44).
    • BorgInfoArchiveJob.process_result dereferences the result of get_or_none without a check (src/vorta/borg/info_archive.py:46-47).
    • Any of these raises DoesNotExist or AttributeError if the repo or archive row was removed or replaced in the meantime, for example by an archive refresh. These are low-probability races.

Scope and risks

  • Size: about 40-60 lines in borg_job.py and jobs_manager.py, plus tests. No changes to the result consumers.
  • The synthetic result must look like a real one. Consumers read result['params'][...], result['returncode'], result['errors'] and result.get('cmd'), which are all safe if those keys are present. result['data'] is read only when returncode == 0, except in VortaApp.set_borg_details_result (src/vorta/application.py:189), which does 'version' in result['data']; this works with '' or {}, and that slot then shows the "Borg not found" alert, which is reasonable.
  • Do not emit app.backup_finished_event for job types that never emit it. The scheduler slot lambda res: self.set_timer_for_profile(res['params']['profile_id']) (src/vorta/scheduler/__init__.py:65) raises KeyError on jobs without profile_id, such as BorgVersionJob.
  • BorgCreateJob.finished_event runs the post-backup command (src/vorta/borg/create.py:63-66). Calling it on the error path runs that command with the error return code. This matches what happens after a failed Borg run, but it is a behaviour change worth noting.
  • BorgCheckJob.finished_event emits check_failed_event for any non-zero return code (src/vorta/borg/check.py:25-35), so a crashed check shows the "Repo check failed" dialog. That is acceptable.
  • Swallowing exceptions risks hiding real bugs. Part 3 is needed for that reason: without the main-thread dialog, crashes that are visible today on a console would turn into quiet "failed" jobs.

Notes for implementation

There are no tests of job failure paths today. tests/unit/test_borg.py:13-25 shows how to run a job with a mocked Popen, and tests/unit/conftest.py:220-233 provides mock_borg_popen.

Add tests for:

  • Popen mocked with side_effect=FileNotFoundError:
    • result is emitted once with a non-success returncode and a non-empty errors.
    • The EventLogModel row has returncode and end_time set.
    • Files in cleanup_files are closed.
  • A stderr line 0 or {} produces a result instead of an exception.
  • A subclass whose finished_event raises still emits result exactly once.
  • JobsManager with two jobs queued for the same site, where the first raises: the second still runs, and is_worker_running() returns False afterwards.
  • Scheduler: a raising BorgCreateJob still reaches SchedulerExecution.notify and records the JobModel row as failed.

Activity

  1. added
    priority:mediumNon-critical improvement to functionality or usability
    type:bugSomething doesn't work as intended
    on Nov 14, 2026
  2. added a commit that references this issue on Nov 15, 2026
    f08f381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:mediumNon-critical improvement to functionality or usabilitytype:bugSomething doesn't work as intended

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions