Skip to content

fix(specs): BREAKING CHANGE - make AB test pValue optional - #7066

Merged
Fluf22 merged 3 commits into
mainfrom
fix/abtesting-v3/optional-pvalue
Sep 21, 2026
Merged

Fluf22 merged 3 commits into
mainfrom
fix/abtesting-v3/optional-pvalue

Conversation

@stevenMevans

@stevenMevans stevenMevans commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

🧭 What and Why

The A/B Testing v3 API has allowed metricResult.pValue to be omitted for some time when no Frequentist result is available. The OpenAPI spec still marks it as required, so generated clients do not accurately represent the existing API behavior and may reject valid responses.

This PR brings the spec and generated public model types back in line with the runtime contract. Bayesian tests exposed the mismatch because Bayesian-only responses omit Frequentist inference fields, but the mismatch predates that work and is not specific to Bayesian testing.

Changes included:

  • Makes metricResult.pValue optional.
  • Documents when the API omits pValue.

This changes generated public model types from required to optional or nullable across client languages and should be handled as a breaking client release.

Changelog

metricResult.pValue is now optional in generated A/B Testing v3 clients, matching API responses where no p-value is available for a metric. This is a breaking type change: code that assumes pValue is always present must handle an absent or nullable value.

πŸ§ͺ Tests

Green CI

@algolia-api-clients-automation-bot

algolia-api-clients-automation-bot Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

πŸ’₯ Breaking changes detected !!

Either this PR or a previous PR not released yet introduced breaking changes, be careful when merging.
You can find the details in the 'client java@21' CI job.

βœ”οΈ Code generated!

Name Link
πŸͺ“ Triggered by 2465b99d139994e6486bc72b41b3d0ebd49ad203
πŸƒ Generated commit e1f9d72ef68dfa4b720e0ccec83df2976470db80
🌲 Generated branch generated/fix/abtesting-v3/optional-pvalue
πŸ“Š Benchmark results

Benchmarks performed on the method using a mock server, the results might not reflect the real-world performance.

Language Req/s
javascript 2229
php 1949
go 1788
csharp 1518
python 1255
java 1065
scala 1049
ruby 868
swift 574

@stevenMevans
stevenMevans force-pushed the feat/abtesting-hypothesis branch from 571a47d to d31d2e7 Compare September 17, 2026 18:36
@stevenMevans
stevenMevans force-pushed the fix/abtesting-v3/optional-pvalue branch from fdaf09f to 8352939 Compare September 17, 2026 18:36
@stevenMevans stevenMevans changed the title fix(specs): make AB test pValue optional fix(specs): BREAKING CHANGE - make AB test pValue optional Sep 17, 2026
@stevenMevans stevenMevans self-assigned this Sep 17, 2026
@stevenMevans stevenMevans added Specs Issue related to the REST API specs and removed Specs Issue related to the REST API specs labels Sep 17, 2026
@stevenMevans
stevenMevans force-pushed the fix/abtesting-v3/optional-pvalue branch 2 times, most recently from 55bd737 to 3f77878 Compare September 17, 2026 22:25
@stevenMevans
stevenMevans requested a balanced review from Copilot September 18, 2026 13:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟒 Approval recommended

The focused schema change correctly represents the documented runtime behavior without introducing inconsistencies.

Pull request overview

Aligns the A/B Testing v3 schema with API responses that may omit pValue.

Changes:

  • Makes metricResult.pValue optional.
  • Documents omission behavior and updates the example.
File summaries
File Description
specs/abtesting-v3/common/schemas/Variant.yml Updates the pValue schema contract and documentation.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Balanced

πŸ’‘ Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@stevenMevans
stevenMevans marked this pull request as ready for review September 18, 2026 14:02
@stevenMevans
stevenMevans requested a review from a team as a code owner September 18, 2026 14:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
πŸ“ Code Review βœ… Completed 2026-09-18T14:05:02.836854Z 3f77878 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with πŸ‘€ while any review is running, comments if it has suggestions, and reacts with πŸ‘ once all reviews finish with no findings.

@stevenMevans
stevenMevans force-pushed the feat/abtesting-hypothesis branch from d31d2e7 to c7daff8 Compare September 18, 2026 14:24
@stevenMevans
stevenMevans force-pushed the fix/abtesting-v3/optional-pvalue branch from 3f77878 to 1543400 Compare September 18, 2026 14:24
@Fluf22
Fluf22 force-pushed the feat/abtesting-hypothesis branch from c7daff8 to 57555d2 Compare September 18, 2026 17:58
@stevenMevans
stevenMevans force-pushed the feat/abtesting-hypothesis branch from 57555d2 to 8154217 Compare September 21, 2026 13:13
@stevenMevans
stevenMevans force-pushed the fix/abtesting-v3/optional-pvalue branch from 1543400 to bb6471b Compare September 21, 2026 13:13
Fluf22
Fluf22 previously approved these changes Sep 21, 2026
Base automatically changed from feat/abtesting-hypothesis to main September 21, 2026 16:04
@stevenMevans
stevenMevans force-pushed the fix/abtesting-v3/optional-pvalue branch from bb6471b to a948787 Compare September 21, 2026 16:05
@aikido-pr-checks

Copy link
Copy Markdown

Summary by Aikido

Security Issues: 0 Quality Issues: 0 Resolved Issues: 0

⚑ Enhancements

  • Increased JavaScript bundle size limit to accommodate generated client changes

More info

@stevenMevans
stevenMevans requested a review from Fluf22 September 21, 2026 17:24
@stevenMevans

Copy link
Copy Markdown
Contributor Author

@Fluf22 thank you for all the reviews! πŸ™ Because the Github stack required rebasing after each PR merge, I've merged the stack into this single feature branch that will need your sign-off one more time.

@stevenMevans
stevenMevans enabled auto-merge (squash) September 21, 2026 17:27
@Fluf22
Fluf22 merged commit b58c408 into main Sep 21, 2026
39 checks passed
@Fluf22
Fluf22 deleted the fix/abtesting-v3/optional-pvalue branch September 21, 2026 17:37
algolia-api-clients-automation-bot Bot added a commit that referenced this pull request Sep 21, 2026
…enerated) [skip ci]

Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-csharp that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-dart that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-go that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-java that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-javascript that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-kotlin that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-php that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-python that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-ruby that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-scala that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
algolia-api-clients-automation-bot Bot added a commit to algolia/algoliasearch-client-swift that referenced this pull request Sep 21, 2026
algolia/api-clients-automation#7066

Co-authored-by: algolia-api-clients-automation-bot[bot] <288895823+algolia-api-clients-automation-bot[bot]@users.noreply.github.com>
Co-authored-by: Steven Evans <steven.evans@algolia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants