Repository navigation
Closed
Labels
priority:mediumNon-critical improvement to functionality or usabilityNon-critical improvement to functionality or usabilitytype:bugSomething doesn't work as intendedSomething doesn't work as intended
Description
Line numbers are as of commit 9c4fc1e.
Problem
If a job raises inside job.run(), nothing reports that the job finished.
SiteWorker.run(src/vorta/borg/jobs_manager.py:52-65) only catchesqueue.Empty. Any other exception leaves the loop, and thethreading.Threaddies.- Only
sys.excepthookis installed (src/vorta/__main__.py:38).threading.excepthookis not set anywhere, so the traceback goes to stderr only. It does not reach the log file, and noExceptionDialogappears. In background mode or from the app bundle, that means it is lost. BorgJob._run(src/vorta/borg/borg_job.py:262-386) has notry/finally. By the timePopenruns (:282),started_event()(:263) has already emittedbackup_started_eventfor most job types, and anEventLogModelrow has been saved (:265-270). If anything raises after that:resultandapp.backup_finished_eventare never emitted (:384), soMainWindow.backup_finished_event(src/vorta/views/main_window.py:339-344) never re-enables the Start Backup and archive buttons.- The tray icon stays in its active state (
src/vorta/application.py:156-161). - The event log row keeps no
returncodeand noend_time(:375-381). cleanup_filesare never closed (:385-386). For extract, these hold the temporary pattern file.- For a scheduled backup,
SchedulerExecution.notifynever runs (src/vorta/scheduler/execution.py:99). The timer was already removed before the job was queued (:96), so that profile is not re-armed by this run. TheJobModelrow staysrunninguntil the next start, whensrc/vorta/store/connection.py:62-64marks itinterrupted. - For post-backup tasks,
_post_backup_pendingnever reaches zero (execution.py:224-246), so the "Post Backup Tasks ..." notification never appears.
- Jobs still queued for that repo are not lost.
is_worker_running()now returns False because the thread is dead (jobs_manager.py:97), and the nextadd_jobfor the same repo starts a newSiteWorkeron the same queue (:122-126). Until another job is added for that repo, though, the queued jobs do not run.
This has happened in practice. #1766 shows Exception in thread Thread-1 from a KeyError in the stderr parsing loop of BorgVersionJob. That particular key was fixed later, but the failure mode is still there.
Proposed change
- In
BorgJob.run/_run, finalise every job even when it raises:- Wrap everything after
started_event()intry/except/finally. - On an exception, call
logger.exception(...)so the traceback reaches the log file. - Build a result with the same keys as a normal one:
params,cmd,returncodeset to a non-success value (for example 2),errors=[(logging.CRITICAL, '<ExceptionType>: <message>')],dataset to{}or'', andlog_entry_id. - Save the event log row with that
returncodeandend_time. - Emit the result only once. Keep a flag that is set when
finished_eventhas been entered. If the exception came fromfinished_eventitself, fall back to emittingself.result, plusapp.backup_finished_eventwhen this job type emits it normally, without callingfinished_eventagain. - Close
cleanup_filesinfinally.
- Wrap everything after
- In
SiteWorker.run, catchExceptionaroundjob.run(), log it, clearcurrent_job, and move on to the next queued job. This is the backstop for anyJobInterfacethat is not aBorgJob. TodayBorgJobis the only implementation insrc/. - Keep bugs visible. Route the exception to the main thread through a queued Qt signal, so the existing
ExceptionDialogand file log still appear (a worker thread must not create the dialog itself). Also setthreading.excepthookto log throughlogger.criticalas a last resort.
Bugs this fixes
Each of these currently leaves the UI blocked as described above. After the change, each one ends as a failed job with a logged traceback. The root causes still need their own fixes.
- Extract to a folder that no longer exists.
BorgExtractJob.preparepassescwd=destination_folder(src/vorta/borg/extract.py:93). If that folder is removed or unmounted before the queued job starts,Popen(borg_job.py:282-291) raisesFileNotFoundErrorafterbackup_started_eventand after the event log row has been saved. The Borg binary disappearing betweenprepare()andrun()(cmd[0]is set atborg_job.py:102) has the same effect. - A stderr line that is valid JSON but not a dict with
type.parsed['type'](borg_job.py:316-318) raisesTypeErrorfor something like0or"x", andKeyErrorfor a dict withouttype. OnlyJSONDecodeErroris caught (:351). This is unlikely with Borg's own--log-jsonoutput, but stderr also carries output from SSH and remote wrappers. - Row removed while the job was queued.
BorgPruneJob.finished_eventandBorgDeleteJob.finished_eventcallRepoModel.get(id=...)before they emitbackup_finished_event(src/vorta/borg/prune.py:14,src/vorta/borg/delete.py:16).BorgRenameJob.process_resultcallsRepoModel.getandArchiveModel.get(src/vorta/borg/rename.py:43-44).BorgInfoArchiveJob.process_resultdereferences the result ofget_or_nonewithout a check (src/vorta/borg/info_archive.py:46-47).- Any of these raises
DoesNotExistorAttributeErrorif the repo or archive row was removed or replaced in the meantime, for example by an archive refresh. These are low-probability races.
Scope and risks
- Size: about 40-60 lines in
borg_job.pyandjobs_manager.py, plus tests. No changes to the result consumers. - The synthetic result must look like a real one. Consumers read
result['params'][...],result['returncode'],result['errors']andresult.get('cmd'), which are all safe if those keys are present.result['data']is read only whenreturncode == 0, except inVortaApp.set_borg_details_result(src/vorta/application.py:189), which does'version' in result['data']; this works with''or{}, and that slot then shows the "Borg not found" alert, which is reasonable. - Do not emit
app.backup_finished_eventfor job types that never emit it. The scheduler slotlambda res: self.set_timer_for_profile(res['params']['profile_id'])(src/vorta/scheduler/__init__.py:65) raisesKeyErroron jobs withoutprofile_id, such asBorgVersionJob. BorgCreateJob.finished_eventruns the post-backup command (src/vorta/borg/create.py:63-66). Calling it on the error path runs that command with the error return code. This matches what happens after a failed Borg run, but it is a behaviour change worth noting.BorgCheckJob.finished_eventemitscheck_failed_eventfor any non-zero return code (src/vorta/borg/check.py:25-35), so a crashed check shows the "Repo check failed" dialog. That is acceptable.- Swallowing exceptions risks hiding real bugs. Part 3 is needed for that reason: without the main-thread dialog, crashes that are visible today on a console would turn into quiet "failed" jobs.
Notes for implementation
There are no tests of job failure paths today. tests/unit/test_borg.py:13-25 shows how to run a job with a mocked Popen, and tests/unit/conftest.py:220-233 provides mock_borg_popen.
Add tests for:
Popenmocked withside_effect=FileNotFoundError:resultis emitted once with a non-successreturncodeand a non-emptyerrors.- The
EventLogModelrow hasreturncodeandend_timeset. - Files in
cleanup_filesare closed.
- A stderr line
0or{}produces a result instead of an exception. - A subclass whose
finished_eventraises still emitsresultexactly once. JobsManagerwith two jobs queued for the same site, where the first raises: the second still runs, andis_worker_running()returns False afterwards.- Scheduler: a raising
BorgCreateJobstill reachesSchedulerExecution.notifyand records theJobModelrow asfailed.
Metadata
Metadata
Assignees
Labels
priority:mediumNon-critical improvement to functionality or usabilityNon-critical improvement to functionality or usabilitytype:bugSomething doesn't work as intendedSomething doesn't work as intended
Activity
fix: always finish a Borg job and unblock the UI when it raises (borgβ¦