Skip to content

feat(python): accept iterators in bulk indexing helpers - #6486

Merged
eric-zaharia merged 16 commits into
mainfrom
feat/python-batch-iterators
Jun 12, 2026
Merged

eric-zaharia merged 16 commits into
mainfrom
feat/python-batch-iterators

Conversation

@eric-zaharia

@eric-zaharia eric-zaharia commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

🧭 What and Why

🎟 JIRA Ticket: API-424

Restores iterator/generator support to the Python bulk-indexing helpers, addressing the v3 β†’ v4 regression reported in algoliasearch-client-python#584. v4 narrowed these helpers to require a materialized List, so users replacing datasets too large to fit in memory could no longer stream their records (v1 explicitly supported iterators for this reason).

Changes included:

  • Accept Iterable[Dict[str, Any]] instead of List[...] on save_objects, save_objects_with_transformation, partial_update_objects, partial_update_objects_with_transformation, chunked_batch, chunked_push, replace_all_objects, and replace_all_objects_with_transformation. Generators are now first-class, so large datasets can be streamed without being held in memory all at once.
  • Single-pass batching. The previous for i, obj in enumerate(objects): … if len(records) == batch_size or i == len(objects) - 1 relied on len(objects), which a generator can't provide. Each helper now flushes the trailing partial batch after the loop (if records: / if requests:), so objects is consumed exactly once.
  • Fixes the chunked_push task-polling offset bug (documented in Confluence: Chunked Push Task-Polling Bug (waitForTasks)).

πŸ§ͺ Test

  • Adds tests/output/python/tests/manual/test_chunked_helpers_iterator.py, which drives chunked_batch, save_objects, and save_objects_with_transformation with a single-pass generator (no __len__) and asserts records are batched in order β€” e.g. 5 records at batch_size=2 produce batches of [2, 2, 1] β€” for both the sync and async clients.
  • Existing CTS helper suites (pushMock, replaceAllObjectsWithTransformation) continue to pass, exercising the refactored chunked_push / chunked_batch paths.

@eric-zaharia eric-zaharia self-assigned this Jun 9, 2026
@algolia-api-clients-automation-bot

algolia-api-clients-automation-bot Bot commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

βœ”οΈ Code generated!

Name Link
πŸͺ“ Triggered by 43ecf39ca9bfd311ccdd46fdc51443c073d273b3
πŸƒ Generated commit eddfce8bc3538855057d061ab0b59a2ce4a12800
🌲 Generated branch generated/feat/python-batch-iterators
πŸ“Š Benchmark results

Benchmarks performed on the method using a mock server, the results might not reflect the real-world performance.

Language Req/s
javascript 4514
go 1937
php 1724
csharp 1436
python 1432
java 1109
ruby 978
swift 828
scala 23

@eric-zaharia
eric-zaharia marked this pull request as ready for review June 10, 2026 13:37
@eric-zaharia
eric-zaharia requested a review from a team as a code owner June 10, 2026 13:37
@eric-zaharia
eric-zaharia enabled auto-merge (squash) June 12, 2026 09:32
@eric-zaharia
eric-zaharia merged commit 480b4ec into main Jun 12, 2026
35 checks passed
@eric-zaharia
eric-zaharia deleted the feat/python-batch-iterators branch June 12, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants