Skip to content

openIfChanged returns null after flush() when the only changes are doc-values updatesΒ #16900

Open
diwu-sf/lucene
#1

Description

@shbhar

Description

If the only changes are doc value updates (including soft deletes) since reader was opened and then flush is called after, DirectoryReader.openIfChanged(reader, writer) returns null and reader.isCurrent() returns true, but reader does not actually see the update (fresh open with DirectoryReader.open(writer) does see it). SearcherManager.maybeRefresh() also doesnt pick up the change due to this.

Repro (passes without the flush()).

  public void testDocValuesUpdateThenFlush() throws Exception {
    try (Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
      Document doc = new Document();
      doc.add(new StringField("id", "1", Field.Store.NO));
      doc.add(new NumericDocValuesField("val", 1));
      w.addDocument(doc);

      try (DirectoryReader r = DirectoryReader.open(w)) {
        w.updateNumericDocValue(new Term("id", "1"), "val", 2);
        w.flush(); // remove this line and the test passes

        DirectoryReader r2 = DirectoryReader.openIfChanged(r, w);
        assertNotNull("openIfChanged missed the doc values update", r2); // fails: r2 == null
        r2.close();
      }
    }
  }

According to AI agent the bug has been there for some time, summary below

A no-op forceMerge(1) has the same effect, and so does a reader opened with DirectoryReader.open(writer, true, true), even without flush(). It works in 7.3.1 and fails from 7.4.0 through 10.5.2 and main.

Likely cause: since LUCENE-8290 (#9337), doFlush writes pending doc-values updates through writeReaderPool(true),
i.e. ReaderPool.commit() then checkpointNoSIS(), which does not bump SegmentInfos.version. The comment in
ReaderPool.commit assumes the bump already happened when the updates were applied, which is true for deletes but
not for doc-values updates. With nothing pending and the version unchanged, nrtIsCurrent() returns true.

umbrella issue for AI found bugs: #16937

Version and environment details

Tested on main b83101c, branch_10_6, 10.5.2, JDKΒ 25, Amazon Linux 2, aarch64

Activity

  1. Roshan1299 commented on Oct 27, 2026

    @Roshan1299

    Hi @shbhar , I'd like to work on this! Can I be assigned?

  2. shbhar commented on Oct 28, 2026

    @shbhar
    ContributorAuthor

    @Roshan1299 sure, please go for it

  3. diwu-sf commented on Oct 28, 2026

    @diwu-sf

    When you fix this, since the fix is to the NRT reader, can you also make sure to cover an adjacent bug:

    A reader opened during a commit. A commit bumps the version first, then syncs files to disk, and only then switches to the new segments_N file. A reader opened during the sync gets the new version but still points at the old segments_N. Because the versions match, it is treated as up to date even after that file is deleted. Fix: also check that the reader points at the latest commit.

    If you look at the PR #16923 it will have the fix + failing test cases.
    This is the root cause of the upstream OpenSearch bug opensearch-project/OpenSearch#11025

    I would prefer for this to be fixed in Lucene vs having to patch around it in OpenSearch. Thanks

  4. Roshan1299 commented on Oct 30, 2026

    @Roshan1299

    @diwu-sf thanks, I hadn't seen #16923. It already covers #16900 and the commit/sync case with tests, so I don't want to open a competing PR with the same fix. Are you planning to take #16923 through review? If so I'll step back on this issue. Happy to help test it if that's useful.

  5. diwu-sf commented on Oct 30, 2026

    @diwu-sf

    Yes I can take this one through review. We needed this primarily because our OpenSearch cluster has hit the NRT reader issue and this is a real annoyance for us until Lucene has it fixed and OpenSearch picks up on the updated lucene.

  6. diwu-sf commented on Oct 31, 2026

    @diwu-sf

    @shbhar can you review #16923 please? thanks

  7. shbhar commented on Nov 3, 2026

    @shbhar
    ContributorAuthor

    Spent some time on it. From my understanding for openIfChanged same version = same reader is a contract, which would be broken, unless we bump the version on commit as well (which should be ok?), more details + test failure repro in the comment

  8. diwu-sf commented on Nov 3, 2026

    @diwu-sf

    Yes you're right, it's easier and more correct to just bump the version.

    Updated the PR and added a bunch more concurrency tests for race conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions