Repository navigation
feat(python): accept iterators in bulk indexing helpers - #6486
Merged
Merged
Conversation
Contributor
βοΈ Code generated!
π Benchmark resultsBenchmarks performed on the method using a mock server, the results might not reflect the real-world performance.
|
Fluf22
approved these changes
Jun 12, 2026
eric-zaharia
enabled auto-merge (squash)
June 12, 2026 09:32
This was referenced Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
π§ What and Why
π JIRA Ticket: API-424
Restores iterator/generator support to the Python bulk-indexing helpers, addressing the v3 β v4 regression reported in algoliasearch-client-python#584. v4 narrowed these helpers to require a materialized
List, so users replacing datasets too large to fit in memory could no longer stream their records (v1 explicitly supported iterators for this reason).Changes included:
Iterable[Dict[str, Any]]instead ofList[...]onsave_objects,save_objects_with_transformation,partial_update_objects,partial_update_objects_with_transformation,chunked_batch,chunked_push,replace_all_objects, andreplace_all_objects_with_transformation. Generators are now first-class, so large datasets can be streamed without being held in memory all at once.for i, obj in enumerate(objects): β¦ if len(records) == batch_size or i == len(objects) - 1relied onlen(objects), which a generator can't provide. Each helper now flushes the trailing partial batch after the loop (if records:/if requests:), soobjectsis consumed exactly once.chunked_pushtask-polling offset bug (documented in Confluence: Chunked Push Task-Polling Bug (waitForTasks)).π§ͺ Test
tests/output/python/tests/manual/test_chunked_helpers_iterator.py, which driveschunked_batch,save_objects, andsave_objects_with_transformationwith a single-pass generator (no__len__) and asserts records are batched in order β e.g. 5 records atbatch_size=2produce batches of[2, 2, 1]β for both the sync and async clients.pushMock,replaceAllObjectsWithTransformation) continue to pass, exercising the refactoredchunked_push/chunked_batchpaths.