You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
openIfChanged returns null after flush() when the only changes are doc-values updatesΒ #16900
If the only changes are doc value updates (including soft deletes) since reader was opened and then flush is called after, DirectoryReader.openIfChanged(reader, writer) returns null and reader.isCurrent() returns true, but reader does not actually see the update (fresh open with DirectoryReader.open(writer) does see it). SearcherManager.maybeRefresh() also doesnt pick up the change due to this.
Repro (passes without the flush()).
publicvoidtestDocValuesUpdateThenFlush() throwsException {
try (Directorydir = newDirectory();
IndexWriterw = newIndexWriter(dir, newIndexWriterConfig())) {
Documentdoc = newDocument();
doc.add(newStringField("id", "1", Field.Store.NO));
doc.add(newNumericDocValuesField("val", 1));
w.addDocument(doc);
try (DirectoryReaderr = DirectoryReader.open(w)) {
w.updateNumericDocValue(newTerm("id", "1"), "val", 2);
w.flush(); // remove this line and the test passesDirectoryReaderr2 = DirectoryReader.openIfChanged(r, w);
assertNotNull("openIfChanged missed the doc values update", r2); // fails: r2 == nullr2.close();
}
}
}
According to AI agent the bug has been there for some time, summary below
A no-op forceMerge(1) has the same effect, and so does a reader opened with DirectoryReader.open(writer, true, true), even without flush(). It works in 7.3.1 and fails from 7.4.0 through 10.5.2 and main.
Likely cause: since LUCENE-8290 (#9337), doFlush writes pending doc-values updates through writeReaderPool(true),
i.e. ReaderPool.commit() then checkpointNoSIS(), which does not bump SegmentInfos.version. The comment in ReaderPool.commit assumes the bump already happened when the updates were applied, which is true for deletes but
not for doc-values updates. With nothing pending and the version unchanged, nrtIsCurrent() returns true.
When you fix this, since the fix is to the NRT reader, can you also make sure to cover an adjacent bug:
A reader opened during a commit. A commit bumps the version first, then syncs files to disk, and only then switches to the new segments_N file. A reader opened during the sync gets the new version but still points at the old segments_N. Because the versions match, it is treated as up to date even after that file is deleted. Fix: also check that the reader points at the latest commit.
@diwu-sf thanks, I hadn't seen #16923. It already covers #16900 and the commit/sync case with tests, so I don't want to open a competing PR with the same fix. Are you planning to take #16923 through review? If so I'll step back on this issue. Happy to help test it if that's useful.
Yes I can take this one through review. We needed this primarily because our OpenSearch cluster has hit the NRT reader issue and this is a real annoyance for us until Lucene has it fixed and OpenSearch picks up on the updated lucene.
Spent some time on it. From my understanding for openIfChanged same version = same reader is a contract, which would be broken, unless we bump the version on commit as well (which should be ok?), more details + test failure repro in the comment
Activity
Roshan1299 commentedon Oct 27, 2026
Hi @shbhar , I'd like to work on this! Can I be assigned?
Make written doc values updates visible to NRT readers
Make written doc values updates visible to NRT readers
Make written doc values updates visible to NRT readers
shbhar commentedon Oct 28, 2026
@Roshan1299 sure, please go for it
diwu-sf commentedon Oct 28, 2026
When you fix this, since the fix is to the NRT reader, can you also make sure to cover an adjacent bug:
If you look at the PR #16923 it will have the fix + failing test cases.
This is the root cause of the upstream OpenSearch bug opensearch-project/OpenSearch#11025
I would prefer for this to be fixed in Lucene vs having to patch around it in OpenSearch. Thanks
Roshan1299 commentedon Oct 30, 2026
@diwu-sf thanks, I hadn't seen #16923. It already covers #16900 and the commit/sync case with tests, so I don't want to open a competing PR with the same fix. Are you planning to take #16923 through review? If so I'll step back on this issue. Happy to help test it if that's useful.
diwu-sf commentedon Oct 30, 2026
Yes I can take this one through review. We needed this primarily because our OpenSearch cluster has hit the NRT reader issue and this is a real annoyance for us until Lucene has it fixed and OpenSearch picks up on the updated lucene.
diwu-sf commentedon Oct 31, 2026
@shbhar can you review #16923 please? thanks
shbhar commentedon Nov 3, 2026
Spent some time on it. From my understanding for openIfChanged same version = same reader is a contract, which would be broken, unless we bump the version on commit as well (which should be ok?), more details + test failure repro in the comment
diwu-sf commentedon Nov 3, 2026
Yes you're right, it's easier and more correct to just bump the version.
Updated the PR and added a bunch more concurrency tests for race conditions.