Skip to content

Fix segfault from racing log level changes with concurrent logging - #15471

Merged
Nir-Az merged 6 commits into
realsenseai:developmentfrom
Binyaminp:fix-log-level-race-segfault
Aug 3, 2026
Merged

Nir-Az merged 6 commits into
realsenseai:developmentfrom
Binyaminp:fix-log-level-race-segfault

Conversation

@Binyaminp

@Binyaminp Binyaminp commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Overview: Fixes a segfault that occurs when rs2::log_to_console() (or log_to_file/log_to_callback/reset_logger) is called from one thread while another thread is logging.

Tracked on [RSDSO-21284]

Why

Reported in GH #14761: changing the log level while a camera streams intermittently segfaults, with crashes seen in el::base::TypedConfigurations::enabled().

The LIBRS_LOG_/LIBRS_LOG_STR_ macros (backing LOG_DEBUG, rs2_log, etc.) checked logger->enabled(level) without holding the logger's lock. Meanwhile, Logger::configure() (invoked by rs2::log_to_console() et al., via EasyLogging++'s el::Loggers::reconfigureLogger()) deletes and replaces the logger's TypedConfigurations object, and separately mutates m_unflushedCount (relevant to the log_to_file path), all without adequate synchronization against a concurrent reader.

Summary

  • Read side (third-party/rsutils/include/rsutils/easylogging/easyloggingpp.h): added rsutils::elpp_enabled(), which checks logger->enabled() while holding the logger's own lock, instead of unlocked. Lock is held only for the check, not while building/streaming the message, avoiding a lock-order-inversion risk.
  • Write side (src/log.h): every call site that triggers Logger::configure() (open(), open_def(), reset_logger(), enable_rolling_log_file()) now acquires that same per-Logger (recursive) lock before calling into el::Loggers::reconfigureLogger(). This fully serializes reconfiguration against concurrent logging without modifying vendored EasyLogging++ code — confirmed no newer upstream release fixes this (the project is archived; its final release, v9.97.1, has the identical bug).
  • Removed a pre-existing, unused std::mutex log_mutex member that sat unused right next to this code.
  • Added unit-tests/log/test-log-level-race.cpp with two regression tests: one via log_to_console (the originally reported path), one via log_to_file (a related race in the same function, found during review).

Test plan

  • Both test cases in unit-tests/log/test-log-level-race.cpp reproduce real SIGSEGVs on the original code and pass cleanly with the fix.
  • Verified on Windows via instrumented crash-hook stack traces: no crash remains in the logging/reconfigure path post-fix (a separate, pre-existing, unrelated Windows-only crash in the test harness's own teardown is out of scope — reproduces identically with zero logging calls).
  • Verified on Linux (Ubuntu 22.04): reverting only src/log.h and rebuilding reliably reproduces Segmentation fault (core dumped); with the fix, 5/5 clean runs (All tests passed, 10 assertions in 2 test cases).

🤖 Generated by AI

@sysrsbuild-gh-agentic sysrsbuild-gh-agentic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Auto-generated review by rs-agentic bot

Comment thread unit-tests/log/test-log-level-race.cpp Resolved
@Binyaminp
Binyaminp requested a review from Nir-Az July 27, 2026 11:05
@Nir-Az
Nir-Az requested review from ashrafk93 and removed request for Nir-Az July 28, 2026 08:04
@ashrafk93

Copy link
Copy Markdown
Collaborator

Thanks for digging into this — I reproduced the reported crash and confirmed your fix addresses it. Tested on Ubuntu 22.04, GCC, Release, -DELPP_THREAD_SAFE:

build test-log-level-race
development 5/5 SIGSEGV (exit 139)
this PR 5/5 pass

Two things I'd like to see addressed before merge.

1. The same crash is still reachable via rs2::log_to_file()

The PR description says the fix covers log_to_file / log_to_callback / reset_logger as well. It doesn't — those all go through the same Logger::configure(), but only the console path ends up safe.

I took your test and changed exactly one thing, log_to_console → log_to_file:

TEST_CASE( "changing log file level while logging from another thread", "[log]" )
{
    std::atomic< bool > stop( false );

    std::thread level_changer( [&]() {
        while( ! stop )
        {
            rs2::log_to_file( RS2_LOG_SEVERITY_DEBUG, "log-file-level-race.log" );
            rs2::log_to_file( RS2_LOG_SEVERITY_ERROR, "log-file-level-race.log" );
        }
    } );

    std::vector< std::thread > loggers;
    for( int i = 0; i < 4; ++i )
        loggers.emplace_back( [&]() {
            while( ! stop )
                rs2::log( RS2_LOG_SEVERITY_DEBUG, "callback finished" );
        } );

    std::this_thread::sleep_for( std::chrono::milliseconds( 2000 ) );
    stop = true;

    level_changer.join();
    for( auto & t : loggers )
        t.join();
}

5/5 SIGSEGV with this PR applied (and also on development, so this is pre-existing rather than something you introduced):

Thread 5 received signal SIGSEGV
#0  el::base::DefaultLogDispatchCallback::dispatch(std::string&&)
#1  el::base::DefaultLogDispatchCallback::handle(el::LogDispatchData const*)
#2  el::base::LogDispatcher::dispatch()
#3  el::base::Writer::triggerDispatch()
#4  rs2_log()

Why the macro lock can't cover it — Logger::configure() mutates m_unflushedCount before it takes the logger lock:

Thread A — rs2::log_to_file()             Thread B — rs2::log(DEBUG)
────────────────────────────────────      ─────────────────────────────────────────
Logger::configure()          :647         LIBRS_LOG_STR_
  m_isConfigured = false;   ← NO LOCK       ScopedLock lock__( logger->lock() )  ← this PR
  initUnflushedCount();     ← NO LOCK       enabled(Debug) → true
    m_unflushedCount.clear()                Writer … triggerDispatch()
    + 8× insert()                             DefaultLogDispatchCallback::dispatch()  :2235
  …                                             logger->isFlushNeeded(level)
  ScopedLock scopedLock(lock());  :656            ++m_unflushedCount.find(level)->second
  ← lock taken only HERE                          ↑ find() == end() mid-clear() → SIGSEGV

Thread B holds the lock the whole time, exactly as this PR intends — but Thread A doesn't hold it for the initUnflushedCount() part, so there's nothing to serialize against. isFlushNeeded() sits behind if (toFile(level)) (easylogging++.cc:2223), which is why a console-only test never reaches it.

Moving the lock that's already there up to be the first statement fixes it:

 void Logger::configure(const Configurations& configurations) {
+  base::threading::ScopedLock scopedLock(lock());
   m_isConfigured = false;  // we set it to false in case if we fail
   initUnflushedCount();
   if (m_typedConfigurations != nullptr) {
     Configurations* c = const_cast<Configurations*>(m_typedConfigurations->configurations());
     if (c->hasConfiguration(Level::Global, ConfigurationType::Filename)) {
       flush();
     }
   }
-  base::threading::ScopedLock scopedLock(lock());
   if (m_configurations != configurations) {

Safe because the flush() inside re-takes the same mutex, which is recursive on every platform we build (std::recursive_mutex under ELPP_USE_STD_THREADING, otherwise PTHREAD_MUTEX_RECURSIVE / CRITICAL_SECTION). With this, both races pass 5/5 and the full non-live suite is green. Worth adding the file-sink case as a second TEST_CASE so it stays covered.

2. Please narrow the lock scope

os__ << __VA_ARGS__ / << STR now evaluate while the logger lock is held, so arbitrary caller code runs inside the critical section. We log constantly while holding internal mutexes (M → L); any streamed expression that takes a mutex gives L → M and a lock-order inversion.

Nothing in the fix needs that — Writer::initializeLogger re-checks enabled() under the same lock, so building the string outside costs at most one wasted ostringstream when a disable races in. A small helper keeps the lock to just the check and also removes the asymmetry where only LIBRS_LOG_ checked typedConfigurations():

namespace rsutils {
inline bool elpp_enabled( el::Logger * logger__, el::Level level )
{
    if( ! logger__ )
        return false;
    el::base::threading::ScopedLock lock__( logger__->lock() );
    return logger__->typedConfigurations() && logger__->enabled( level );
}
}

then if( rsutils::elpp_enabled( logger__, el::Level::LEVEL ) ) in both macros. I verified this alone still fixes the reported crash (5/5 pass), and combined with the configure() change both races pass 5/5.

(Perf isn't the concern either way, for the record — el::Loggers::getLogger() already takes the registry lock on every log call at easylogging++.cc:1894, so this path was never lock-free.)

Minor

  • The test asserts nothing — Catch2 reports assertions: - none -; it only "passes" by not crashing. Worth a REQUIRE_NOTHROW or an assertion on observed output.
  • "Severity is kept below WARN so nothing actually gets printed to the console" isn't accurate — I see 9–1451 stray DEBUG lines on stdout per run. Level changes are still not atomic from a logging thread's point of view (an ELPP config-merge artifact, separate from the crash and fine to leave out of scope), but the comment and the "clean output" claim in the description should be corrected, and as written the test dumps a lot of noise into CI logs.
  • A couple of continuation backslashes no longer line up with the macro's column, and the new comment block is 5 lines where the guidelines ask for 1–3.
  • Branch is 5 commits behind development.

@Binyaminp
Binyaminp force-pushed the fix-log-level-race-segfault branch from b78c80f to 511daf1 Compare July 28, 2026 12:16
@Binyaminp

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review — both points were real bugs, fixed:

1. log_to_file race — confirmed, and your diagnosis was exactly right: Logger::configure() mutated m_unflushedCount before taking its own lock. Moved the ScopedLock to the first statement in configure() (safe, since flush() re-enters the same recursive mutex). Added your log_to_file scenario as a second TEST_CASE. Verified via an instrumented crash-hook: 5/5 clean runs, both test cases, 10/10 assertions passing.

2. Lock scope — agreed on the inversion risk. Replaced the wrap-everything ScopedLock with your suggested rsutils::elpp_enabled() helper (lock held only for the check), fixing the LIBRS_LOG_/LIBRS_LOG_STR_ asymmetry in the process.

Also addressed: corrected the stray-DEBUG-lines comment, added REQUIRE_NOTHROW around the joins, fixed backslash alignment, trimmed the comment block to 2 lines, and rebased onto current development.

Pushed in 511daf1.

@sysrsbuild-gh-agentic

Copy link
Copy Markdown
Collaborator

🤖 Auto-generated review by rs-agentic bot

Reviewed new commit 511daf1 — no new bugs found.

easylogging++.cc: Moving ScopedLock to the top of configure() is correct — initUnflushedCount() and flush() are now protected by the logger lock, closing the log_to_file data race. The recursive-mutex assumption (per ELPP design) keeps flush() deadlock-free under that lock.

easyloggingpp.h: rsutils::elpp_enabled() is a clean RAII wrapper — no issues.

Minor note: .//log-level-race-file.log double-slash in the file-race test is harmless but looks like a typo; ./log-level-race-file.log is cleaner.

@Binyaminp
Binyaminp force-pushed the fix-log-level-race-segfault branch from 511daf1 to 764a9d5 Compare July 30, 2026 11:36
ashrafk93
ashrafk93 previously approved these changes Jul 30, 2026
}

void Logger::configure(const Configurations& configurations) {
base::threading::ScopedLock scopedLock(lock());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, this is a third party code.
Please check:

  1. Any new version fixed this?
  2. Is it a must? is there something else we can do on our code that will WA this w/o changing third party code?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Binyaminp resolved?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, logic has moved to log.h

@sysrsbuild-gh-agentic

Copy link
Copy Markdown
Collaborator

🤖 rs-agentic bot — synchronize review

New commit (7c85a7c): adds pre-test std::remove(log_file_path) for isolation — looks good, no new issues.

Note: the open BUG comment on line 52 (double slash in .//log-level-race-file.log) is still unresolved in this commit.

Comment thread src/log.h Resolved
Comment thread src/log.h Resolved
@Binyaminp

Copy link
Copy Markdown
Contributor Author

@Nir-Az good catch, and good questions.

1. Newer version? No. Checked upstream (github.com/amrayn/easyloggingpp) — the repo is archived (read-only) and its final release, v9.97.1 (2023), has the exact same unprotected initUnflushedCount() call before the lock in Logger::configure(). This bug is present in every version of this now-dead project we could find, including the last one ever published.

2. Workaround without touching third-party code? Yes, and I've replaced the vendored-file patch with exactly that. Reverted third-party/easyloggingpp/src/easylogging++.cc back to its original, untouched state.

The fix now lives entirely in our own logger_type wrapper (src/log.h): every call site that triggers Logger::configure() (open(), open_def(), the four calls in reset_logger(), enable_rolling_log_file()) is wrapped in a small helper that acquires el::Loggers::getLogger(log_id)->lock() before calling into el::Loggers::reconfigureLogger(...). Since that's the same (recursive) per-Logger mutex that rsutils::elpp_enabled() already holds on the read side, this fully serializes writer against reader — including the unprotected initUnflushedCount() — without needing Logger::configure() itself to change at all.

Also found and removed a pre-existing, entirely unused std::mutex log_mutex member sitting right next to this code (declared, never locked/unlocked anywhere) — presumably an earlier, incomplete attempt at exactly this kind of protection.

Re-verified on Ubuntu 22.04 (perclnx466): reverting only src/log.h and rebuilding gives 3/5 real Segmentation fault (core dumped) runs; with the fix, 5/5 clean (All tests passed, 10 assertions in 2 test cases). Pushed in 4abc9af.

@Binyaminp

Copy link
Copy Markdown
Contributor Author

Re the bot's UNC concern on `.//log-level-race-file.log`: verified empirically on Windows — it resolves to a normal local path (`os.path.abspath` → `C:...\log-level-race-file.log`), not a UNC path. UNC detection requires the path to start with `\`/`//`; the leading `.` here rules that out. This also matches the existing convention already used by `test-multi-thread-logs.cpp` (`.//multi-thread-logs.log`), which ashrafk93 flagged as fine to leave for the same reason. Leaving as-is.

@Binyaminp

Binyaminp commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor Author

@ashrafk93 thanks for verifying, and for the cleanup note — added std::remove(log_file_path) at the end of the file-race test too (7c85a7c). Re-verified on Ubuntu 22.04: the file no longer sticks around after the run (ls now reports "No such file or directory" where it previously left a multi-MB log behind). The .// double-slash point stands as you said — see my reply to the bot's comment on the same thing, confirmed empirically it's not a UNC risk.

@Binyaminp

Copy link
Copy Markdown
Contributor Author

Both fixed:

  1. Null-check on getLogger() — added. In practice log_id is our fixed, always-valid LIBREALSENSE_ELPP_ID constant, so Logger::isValidId() (the only path that makes getLogger() return null when registerIfNotAvailable=true) can't actually fail here — but it's a one-line guard against a documented API contract, cheap enough to just add rather than rely on that invariant silently.
  2. reset_logger() atomicity — batched all four reconfigureLogger calls under a single reconfigure_locked() invocation instead of four separate lock/unlock cycles. A concurrent reader can no longer observe a partially-reset logger.

Re-verified on Ubuntu 22.04: 5/5 clean (All tests passed, 10 assertions in 2 test cases). Pushed in ec0259a.

@Nir-Az
Nir-Az merged commit 2adbcf8 into realsenseai:development Aug 3, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants