You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When encoding a batch of documents the monitor effectively flattens the terms so you lose information about which document(s) a term belongs to. Unfortunately, this can work against the main optimization within the multi-pass term filtered presearcher.
To give an example consider the query (apple AND pear) with df=field. The multi-pass pre-searcher may expand this to {field_0:apple, field_1:pear}. When a document batch comes in [{field:apple}, {field:pear}] the terms get flattened into (field_0:apple OR field_0:pear) AND (field_1:apple OR field_1:pear) which matches the indexed query but is a false positive as the query doesn't match any of the documents in the batch. Now had you sent both documents individually, i.e. (field_0:apple) AND (field_1:apple) and (field_0:pear) AND (field_1:pear), the pre-search would have correctly filtered them out. So batching is often worse from pre-filtering perspective.
Note that this does not affect regular term-filtered presearcher as it always will index one of the branches in the conjunction, i.e field:apple. Which means one of those documents in the hypothetical batch would be false-positive anyway had you sent them individually. This means the false positive candidate query would be run the same amount of times in either case (batched or unbatched).
I think the Presearcher should expose an option to pre-search with each document individually but match against the full batch to capture the efficiency gain on both ends. The Monitor can decide which Presearcher API to invoke depending on what kind of Presearcher it is and/or with what a power-user specifies. It could also be based on batch heuristics, i.e. how correlated the documents are. If the documents are very correlated the negative effect is less noticeable.
Activity