Repository navigation
Fix segfault from racing log level changes with concurrent logging - #15471
Conversation
sysrsbuild-gh-agentic
left a comment
There was a problem hiding this comment.
🤖 Auto-generated review by rs-agentic bot
|
Thanks for digging into this — I reproduced the reported crash and confirmed your fix addresses it. Tested on Ubuntu 22.04, GCC, Release,
Two things I'd like to see addressed before merge. 1. The same crash is still reachable via
|
b78c80f to
511daf1
Compare
|
Thanks for the thorough review — both points were real bugs, fixed: 1. 2. Lock scope — agreed on the inversion risk. Replaced the wrap-everything Also addressed: corrected the stray-DEBUG-lines comment, added Pushed in 511daf1. |
|
🤖 Auto-generated review by rs-agentic bot Reviewed new commit
Minor note: |
511daf1 to
764a9d5
Compare
| } | ||
|
|
||
| void Logger::configure(const Configurations& configurations) { | ||
| base::threading::ScopedLock scopedLock(lock()); |
There was a problem hiding this comment.
Wait, this is a third party code.
Please check:
- Any new version fixed this?
- Is it a must? is there something else we can do on our code that will WA this w/o changing third party code?
There was a problem hiding this comment.
yes, logic has moved to log.h
|
🤖 rs-agentic bot — synchronize review New commit (7c85a7c): adds pre-test Note: the open BUG comment on line 52 (double slash in |
|
@Nir-Az good catch, and good questions. 1. Newer version? No. Checked upstream (github.com/amrayn/easyloggingpp) — the repo is archived (read-only) and its final release, v9.97.1 (2023), has the exact same unprotected 2. Workaround without touching third-party code? Yes, and I've replaced the vendored-file patch with exactly that. Reverted The fix now lives entirely in our own Also found and removed a pre-existing, entirely unused Re-verified on Ubuntu 22.04 (perclnx466): reverting only |
|
Re the bot's UNC concern on `.//log-level-race-file.log`: verified empirically on Windows — it resolves to a normal local path (`os.path.abspath` → `C:...\log-level-race-file.log`), not a UNC path. UNC detection requires the path to start with `\`/`//`; the leading `.` here rules that out. This also matches the existing convention already used by `test-multi-thread-logs.cpp` (`.//multi-thread-logs.log`), which ashrafk93 flagged as fine to leave for the same reason. Leaving as-is. |
|
@ashrafk93 thanks for verifying, and for the cleanup note — added |
|
Both fixed:
Re-verified on Ubuntu 22.04: 5/5 clean ( |
Overview: Fixes a segfault that occurs when
rs2::log_to_console()(orlog_to_file/log_to_callback/reset_logger) is called from one thread while another thread is logging.Tracked on [RSDSO-21284]
Why
Reported in GH #14761: changing the log level while a camera streams intermittently segfaults, with crashes seen in
el::base::TypedConfigurations::enabled().The
LIBRS_LOG_/LIBRS_LOG_STR_macros (backingLOG_DEBUG,rs2_log, etc.) checkedlogger->enabled(level)without holding the logger's lock. Meanwhile,Logger::configure()(invoked byrs2::log_to_console()et al., via EasyLogging++'sel::Loggers::reconfigureLogger()) deletes and replaces the logger'sTypedConfigurationsobject, and separately mutatesm_unflushedCount(relevant to thelog_to_filepath), all without adequate synchronization against a concurrent reader.Summary
third-party/rsutils/include/rsutils/easylogging/easyloggingpp.h): addedrsutils::elpp_enabled(), which checkslogger->enabled()while holding the logger's own lock, instead of unlocked. Lock is held only for the check, not while building/streaming the message, avoiding a lock-order-inversion risk.src/log.h): every call site that triggersLogger::configure()(open(),open_def(),reset_logger(),enable_rolling_log_file()) now acquires that same per-Logger(recursive) lock before calling intoel::Loggers::reconfigureLogger(). This fully serializes reconfiguration against concurrent logging without modifying vendored EasyLogging++ code — confirmed no newer upstream release fixes this (the project is archived; its final release, v9.97.1, has the identical bug).std::mutex log_mutexmember that sat unused right next to this code.unit-tests/log/test-log-level-race.cppwith two regression tests: one vialog_to_console(the originally reported path), one vialog_to_file(a related race in the same function, found during review).Test plan
unit-tests/log/test-log-level-race.cppreproduce realSIGSEGVs on the original code and pass cleanly with the fix.src/log.hand rebuilding reliably reproducesSegmentation fault (core dumped); with the fix, 5/5 clean runs (All tests passed, 10 assertions in 2 test cases).🤖 Generated by AI