Skip to content

Fix: issue 3231 retry infinite loop - #3251

Merged
krmahadevan merged 34 commits into
testng-team:masterfrom
baflQA:fix/issue-3231-retry-infinite-loop
Jan 8, 2026
Merged

krmahadevan merged 34 commits into
testng-team:masterfrom
baflQA:fix/issue-3231-retry-infinite-loop

Conversation

@baflQA

@baflQA baflQA commented Dec 30, 2025 •

Copy link
Copy Markdown
Contributor

Fixes #3231.

Did you remember to?

  • Add test case(s)
  • Update CHANGES.txt
  • Auto applied styling via ./gradlew autostyleApply

Description

This pull request fixes an issue where TestNG's RetryAnalyzer could enter an infinite loop if a test method using a DataProvider mutated the parameter objects before failing.

The Problem

Previously, BaseTestMethod identified test instances for retry analysis using a key generated from the parameter values' hashCode() and equals(). If a test mutated these parameters (e.g., changing an ID field used in hashCode), the next retry attempt would be seen as a new test instance. This caused TestNG to instantiate a fresh RetryAnalyzer for every attempt, resetting the retry counter and leading to an infinite loop. Additionally, different DataProvider rows with identical parameter values would share the same RetryAnalyzer instance, causing them to incorrectly share a retry limit.

The Fix

  1. Stable Identification: Added getParameterIndex() to the ITestResult interface. This index is uniquely and stably assigned by the MethodRunner for each row of a DataProvider.
  2. BaseTestMethod Update: Refactored BaseTestMethod.parameterId() to use ITestResult.getParameterIndex() instead of the parameter-based IdentifiableArrayObject. This ensures the identification key remains constant for a specific test invocation, even if its parameters are mutated.
  3. Independent Retries: Since the key is now based on the row index, different rows with equal parameters now correctly get their own independent RetryAnalyzer instances and counters.

Changes

  • testng-core-api: Added getParameterIndex() to ITestResult.
  • testng-runner-api: Updated TestResult to implement and override new methods correctly.
  • testng-core:
    • Updated BaseTestMethod to use parameterIndex for retry analyzer lookup.
    • Removed unused IdentifiableArrayObject related logic from BaseTestMethod.
    • Added Issue3231Test to verify the fix for mutated parameters.
    • Added RetryLimitTest to verify independent retry limits for identical parameters.
    • Moved "design-to-fail" sample classes to a separate samples package to prevent them from causing build failures in the main suite run.

Summary by CodeRabbit

  • New Features

    • Exposed test parameter index via a new public getter.
  • Bug Fixes

    • Fixed infinite retry loop for mutated data-provider objects.
    • Ensured data-provider parameters refresh when caching is disabled.
    • Corrected false positives in order-insensitive collection comparisons.
    • Fixed ClassCastException related to global thread-pool usage.
    • Restored suite property persistence and improved timeout stack traces.
    • CI: added JDK 25 and JDK 26 EA to test matrix.
  • Tests

    • Added tests and samples covering retries, invocation counts, and per-row retry limits.

✏️ Tip: You can customize this high-level summary in your review settings.

Bartek Florczak and others added 15 commits December 29, 2025 21:00
…false

During test retries, if cacheDataForTestRetries was set to false, the DataProvider was re-executed but the new parameters were ignored. This change ensures that the new parameters are correctly used.
This test confirms that when cacheDataForTestRetries=false, the test method receives the fresh data generated by the DataProvider, rather than cached stale data.
Split tests into runners extending SimpleBaseTest and sample test cases, ensuring proper isolation and style consistency.
…oker.java

Co-authored-by: Julien Herr <julien@herr.fr>
Bumps the Gradle wrapper validation GitHub Action from v3 to v5 for improved security and compatibility.
…ue 3231)

- Add getParameterIndex() to ITestResult interface.
- Use parameterIndex in BaseTestMethod to identify test instances for RetryAnalyzer.
- This ensures a stable key even if parameters are mutated.
- Also fixes shared retry limit for rows with equal parameters.
- Add regression tests Issue3231Test and RetryLimitTest.
@coderabbitai

coderabbitai Bot commented Dec 30, 2025 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@baflQA has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 16 minutes and 5 seconds before requesting another review.

βŒ› How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 66ce00d and 58cedf1.

πŸ“’ Files selected for processing (1)
  • testng-core-api/src/main/java/org/testng/ITestResult.java
πŸ“ Walkthrough

Walkthrough

Replaces object-identity caching for DataProvider parameters with index-based tracking: adds ITestResult.getParameterIndex(), removes IdentifiableArrayObject and per-instance parameter cache, updates BaseTestMethod/TestResult/TestInvoker to use parameter index, adds retry-related tests/samples, and updates CHANGES.txt.

Changes

Cohort / File(s) Summary
API Addition
testng-core-api/src/main/java/org/testng/ITestResult.java
Added default method int getParameterIndex() returning -1.
Internal parameter/tracking removal
IObject & BaseTestMethod
testng-core/src/main/java/org/testng/internal/IObject.java, testng-core/src/main/java/org/testng/internal/BaseTestMethod.java
Removed IdentifiableArrayObject and per-instance parameter cache; parameterId now computed from itr.getParameterIndex() (made static); parameterisation checks adjusted.
TestResult & parameter wiring
testng-runner-api/src/main/java/org/testng/internal/TestResult.java
Added m_parameterIndex (initialized -1), factory TestResult.newTestResult(parameters, index), initParameterIndex(int), deprecated no-op setParameterIndex(int), and annotated several methods with @Override.
Invoker usage
testng-core/src/main/java/org/testng/internal/invokers/TestInvoker.java
Replaced manual creation + setters with TestResult.newTestResult(parameters, parameterIndex) when constructing results for invocations.
Regression tests
testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
Added tests: testRetryWithInvocationCount(), testMutationDoesNotCauseInfiniteRetryLoop(), testEachRowHasItsOwnRetryLimit() to validate retry behavior with mutated parameters and per-row retry limits.
Test samples / fixtures
testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java, .../RetryLimitSample.java, .../InvocationCountRetrySample.java
New sample classes demonstrating mutated DataProvider objects, per-row retry limits, and invocation-count interactions with IRetryAnalyzer implementations.
Changelog
CHANGES.txt
Added 7.12.0 entries documenting fixes for GITHUB-3231, GITHUB-3236, GITHUB-3227, GITHUB-3177, GITHUB-3179, GITHUB-2765 and updated CI JDK matrix (JDK 25/26 EA).

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant DP as DataProvider
  participant Runner as TestNG Runner
  participant Base as BaseTestMethod
  participant TR as TestResult
  participant Test as Test Method
  participant Retry as RetryAnalyzer

  Note over DP,Runner: initial invocation (row N)
  Runner->>DP: obtain parameters (row N)
  DP-->>Runner: parameter object (row N)
  Runner->>Base: invoke test with parameters
  Base->>TR: create via TestResult.newTestResult(params, N)
  TR-->>Base: stores parameter index N
  Base->>Test: execute
  Test-->>Runner: failure
  Runner->>Retry: shouldRetry(TR)
  Retry-->>Runner: decision (reads TR.getParameterIndex())
  alt retry allowed
    Runner->>Base: schedule retry with same parameter index N
  else no retry
    Runner->>TR: mark final failure
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • krmahadevan
  • juherr

Poem

🐰 I nibbled bytes beneath the moonlit test,
Each row now numbered, no more looping quest.
When objects shift and counters stray,
Indexes guide retries back on the way.
Hooray β€” the burrowed bugs are blessed!

πŸš₯ Pre-merge checks | βœ… 4 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.14% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
βœ… Passed checks (4 passed)
Check name Status Explanation
Description Check βœ… Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check βœ… Passed The title clearly and specifically identifies the primary change: fixing issue 3231 related to infinite retry loops, matching the core objective of the PR.
Linked Issues check βœ… Passed The PR addresses all coding requirements from issue #3231: adds getParameterIndex() to ITestResult, updates TestResult to implement it, refactors BaseTestMethod to use parameter index instead of object identity for retry identification, includes comprehensive tests for mutation and independent retry limits, and ensures stable per-invocation identification preventing infinite loops.
Out of Scope Changes check βœ… Passed All changes are directly related to fixing the infinite retry loop issue: core API additions (getParameterIndex), test infrastructure refactors (removing IdentifiableArrayObject), test implementations (three new sample test classes), and documentation updates (CHANGES.txt) are all in scope.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❀️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@baflQA

baflQA commented Dec 30, 2025

Copy link
Copy Markdown
Contributor Author

@coderabbitai generate docstrings

@coderabbitai

coderabbitai Bot commented Dec 30, 2025

Copy link
Copy Markdown
βœ… Actions performed

Initiated docstring generation; will generate only if new commits exist.

@coderabbitai

coderabbitai Bot commented Dec 30, 2025 •

Copy link
Copy Markdown

Note

Docstrings generation - SUCCESS
Generated docstrings for this pull request at #3252

coderabbitai Bot added a commit that referenced this pull request Dec 30, 2025
Docstrings generation was requested by @baflQA.

* #3251 (comment)

The following files were modified:

* `testng-core-api/src/main/java/org/testng/ITestResult.java`
* `testng-core/src/main/java/org/testng/internal/BaseTestMethod.java`
* `testng-core/src/test/java/test/retryAnalyzer/issue3231/samples/MutationSample.java`
* `testng-core/src/test/java/test/retryAnalyzer/issue3231/samples/RetryLimitSample.java`
* `testng-runner-api/src/main/java/org/testng/internal/TestResult.java`
IObject.IdentifiableArrayObject parameter =
new IObject.IdentifiableArrayObject(itr.getParameters());
return parameters.computeIfAbsent(parameter, Function.identity()).getInstanceId();
return Integer.toString(itr.getParameterIndex());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krmahadevan I let you review this part as you made most of it if I remember well.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baflQA - Can you please help check this combination which is what prompted me to add this logic. The commit message would give you the additional context

3505e4b

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Results:

  • Command: ./gradlew :testng-core:test --tests test.retryAnalyzer.RetryAnalyzerTest.ensureRetryDoesntRunEndlesslyForDataDrivenTests -Ptestng.test.extra.jvmargs="-Duser.timezone=America/New_York -Duser.country=RU -Duser.language=ru
    -XX:+UnlockExperimentalVMOptions -XX:hashCode=2"
  • Outcome: BUILD SUCCESSFUL (1/1 tests passed)

All good i think?

@baflQA

baflQA commented Jan 4, 2026

Copy link
Copy Markdown
Contributor Author

Hey @krmahadevan, can i count on Your review?

IObject.IdentifiableArrayObject parameter =
new IObject.IdentifiableArrayObject(itr.getParameters());
return parameters.computeIfAbsent(parameter, Function.identity()).getInstanceId();
return Integer.toString(itr.getParameterIndex());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baflQA - Can you please help check this combination which is what prompted me to add this logic. The commit message would give you the additional context

3505e4b

String id();

/** @return - The index of the parameter row when a @DataProvider is being used. */
default int getParameterIndex() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic relies on the parameter index which is mutable. So that can cause the internal state to change and behavior become unpredictable. I think we would need to ensure that we dont allow that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific suggestion?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made some proposal

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java (1)

51-86: LGTM! Well-structured regression tests for issue #3231.

Both test methods correctly verify the fix:

  • testMutationDoesNotCauseInfiniteRetryLoop validates that mutating data provider objects no longer causes infinite retries.
  • testEachRowHasItsOwnRetryLimit confirms that data provider rows with identical parameter values maintain independent retry counters.

The use of IInvokedMethodListener to count actual invocations is a solid approach, and the expected counts (4 in both cases) align with the retry analyzer configurations.

Optional: Extract common listener pattern

The two test methods share nearly identical listener setup. Consider extracting a helper method to reduce duplication:

private int runTestAndCountInvocations(Class<?> testClass) {
  AtomicInteger invCount = new AtomicInteger(0);
  TestNG tng = create(testClass);
  tng.addListener(
      new IInvokedMethodListener() {
        @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
          if (method.isTestMethod()) {
            invCount.incrementAndGet();
          }
        }
      });
  tng.run();
  return invCount.get();
}

Then simplify the tests:

@Test(description = "GITHUB-3231")
public void testMutationDoesNotCauseInfiniteRetryLoop() {
  MutationSample.guardCounter.set(0);
  assertThat(runTestAndCountInvocations(MutationSample.class)).isEqualTo(4);
}

@Test(description = "GITHUB-3231")
public void testEachRowHasItsOwnRetryLimit() {
  assertThat(runTestAndCountInvocations(RetryLimitSample.class)).isEqualTo(4);
}
testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java (1)

11-19: LGTM! Retry logic is correct.

The MyRetry implementation allows exactly 1 retry (2 total attempts per data provider row), which correctly tests that identical parameter values receive independent retry counters.

Note: This implementation uses increment-then-check style (count++; return count < 2), while MutationSample.MyRetry (lines 49-51 in MutationSample.java) uses check-then-increment style. Both are correct but stylistically inconsistent. Consider standardizing if adding more retry analyzers in the future.

Based on learnings, hardcoding retry limits in test code is acceptable for the TestNG project.

testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java (1)

64-71: LGTM! Test correctly validates the mutation scenario.

The guard counter (line 66) is a defensive safety mechanism to prevent infinite loops during test development. The mutation on line 69 reproduces the original issue where changing parameter object identity would reset retry counters.

Optional: Add explicit guardCounter assertion

Consider asserting the guard counter value in RetryAnalyzerTest.testMutationDoesNotCauseInfiniteRetryLoop() to make the test more explicit:

@Test(description = "GITHUB-3231")
public void testMutationDoesNotCauseInfiniteRetryLoop() {
  MutationSample.guardCounter.set(0);
  AtomicInteger invCount = new AtomicInteger(0);
  // ... listener setup and run ...
  assertThat(invCount.get()).isEqualTo(4);
  assertThat(MutationSample.guardCounter.get()).isEqualTo(4); // Guard never triggered
}

This would explicitly verify that the fix prevents infinite loops rather than relying on the guard's implicit timeout.

πŸ“œ Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between dbcc1d9 and 40c8d9a.

πŸ“’ Files selected for processing (3)
  • testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java
🧰 Additional context used
🧠 Learnings (3)
πŸ““ Common learnings
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:6-9
Timestamp: 2024-10-09T09:04:04.750Z
Learning: In test code for the TestNG project, it's acceptable to hardcode parameters like `retryLimit` since they don't need to be configurable.
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:6-9
Timestamp: 2024-09-30T05:21:21.304Z
Learning: In test code for the TestNG project, it's acceptable to hardcode parameters like `retryLimit` since they don't need to be configurable.
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:11-16
Timestamp: 2024-10-09T09:04:04.750Z
Learning: The `retry(ITestResult result)` method must retain the `ITestResult` parameter because it implements the `IRetryAnalyzer` interface, and method signatures cannot be altered when implementing interfaces, even if the parameter is unused.
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:11-16
Timestamp: 2024-09-30T05:21:00.298Z
Learning: The `retry(ITestResult result)` method must retain the `ITestResult` parameter because it implements the `IRetryAnalyzer` interface, and method signatures cannot be altered when implementing interfaces, even if the parameter is unused.
πŸ“š Learning: 2024-09-30T05:21:21.304Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:6-9
Timestamp: 2024-09-30T05:21:21.304Z
Learning: In test code for the TestNG project, it's acceptable to hardcode parameters like `retryLimit` since they don't need to be configurable.

Applied to files:

  • testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java
πŸ“š Learning: 2024-09-30T05:21:00.298Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:11-16
Timestamp: 2024-09-30T05:21:00.298Z
Learning: The `retry(ITestResult result)` method must retain the `ITestResult` parameter because it implements the `IRetryAnalyzer` interface, and method signatures cannot be altered when implementing interfaces, even if the parameter is unused.

Applied to files:

  • testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java
🧬 Code graph analysis (2)
testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java (1)
testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java (1)
  • MyRetry (43-55)
testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java (2)
testng-core/src/test/kotlin/org/testng/internal/samples/classhelper/issue1339/BabyPandaSample.kt (2)
  • equals (8-17)
  • hashCode (19-19)
testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java (1)
  • MyRetry (11-19)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: 17, temurin, ubuntu, America/New_York, tr_TR, stress JIT
  • GitHub Check: 11, temurin, windows, UTC, tr_TR, stress JIT
  • GitHub Check: 11, corretto, macos, Pacific/Chatham, ru_RU, stress JIT
  • GitHub Check: 26, oracle, macos, UTC, ru_RU
  • GitHub Check: 21, liberica, ubuntu, UTC, ru_RU, stress JIT
  • GitHub Check: 25, zulu, same hashcode, windows, America/New_York, fr_FR
  • GitHub Check: 26, oracle, macos, Pacific/Chatham, ru_RU, stress JIT
πŸ”‡ Additional comments (4)
testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java (2)

21-24: LGTM! Data provider correctly tests identical parameters.

Returning two rows with identical values "a" is intentional and effectively tests that parameter value equality doesn't cause retry analyzer instances to be shared between different data provider rowsβ€”the core issue this PR addresses.


26-29: LGTM! Test method is appropriately simple.

The test correctly fails unconditionally to trigger the retry analyzer and validate the fix.

testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java (2)

14-41: LGTM! Standard mutable object implementation.

The Custom class correctly implements equals() and hashCode() based on the mutable id field using Objects utilities. This implementation is key to reproducing the original issueβ€”mutating the id changes the object's identity, which previously caused retry counters to reset.


43-55: LGTM! Retry analyzer correctly configured.

The MyRetry implementation allows 3 retries (4 total attempts), which aligns with the test expectation in RetryAnalyzerTest.testMutationDoesNotCauseInfiniteRetryLoop() (line 66).

Based on learnings, hardcoding retry limits in test code is acceptable for the TestNG project.

@juherr juherr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you fix style issues?

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java (3)

52-59: Consider adding a test description for consistency.

The test logic is correct (2 invocations Γ— 2 attempts each = 4 total). However, the other new tests (lines 61, 79) include a description attribute. Consider adding one here for consistency, such as @Test(description = "GITHUB-3231").


61-77: LGTM! The mutation test correctly prevents infinite retry loops.

The test properly verifies that parameter mutation doesn't cause infinite retries. The expected count (4 = 1 original + 3 retries) is correct based on MutationSample's retry configuration.

Optional: Verify the guardCounter safety mechanism

Consider adding an assertion to confirm the guardCounter didn't hit its safety limit:

     tng.run();
     assertThat(invCount.get()).isEqualTo(4);
+    assertThat(MutationSample.guardCounter.get()).isEqualTo(4);

This would verify that the safety mechanism (which returns early if counter > 100) was never triggered, providing additional confidence that the fix works correctly.


79-96: LGTM! This test correctly verifies independent retry limits per DataProvider row.

The test validates a critical aspect of the fix: distinct DataProvider rows with identical parameter values receive independent RetryAnalyzer instances. The expected count (4 = 2 rows Γ— 2 attempts each) is correct.

Optional: Reduce listener duplication

The listener setup at lines 83-91 is nearly identical to lines 66-74. If desired, you could extract a helper method:

private IInvokedMethodListener createInvocationCounter(AtomicInteger counter) {
  return new IInvokedMethodListener() {
    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
      if (method.isTestMethod()) {
        counter.incrementAndGet();
      }
    }
  };
}

Then use: tng.addListener(createInvocationCounter(invCount));

However, the current approach is clear and acceptable in test code.

πŸ“œ Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 0aac734 and 849bb38.

πŸ“’ Files selected for processing (3)
  • testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/InvocationCountRetrySample.java
  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
🚧 Files skipped from review as they are similar to previous changes (1)
  • testng-core/src/test/java/test/retryAnalyzer/issue3231/InvocationCountRetrySample.java
🧰 Additional context used
🧠 Learnings (2)
πŸ“š Learning: 2024-09-30T05:21:21.304Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:6-9
Timestamp: 2024-09-30T05:21:21.304Z
Learning: In test code for the TestNG project, it's acceptable to hardcode parameters like `retryLimit` since they don't need to be configurable.

Applied to files:

  • testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
πŸ“š Learning: 2024-10-09T09:04:04.750Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:11-16
Timestamp: 2024-10-09T09:04:04.750Z
Learning: The `retry(ITestResult result)` method must retain the `ITestResult` parameter because it implements the `IRetryAnalyzer` interface, and method signatures cannot be altered when implementing interfaces, even if the parameter is unused.

Applied to files:

  • testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java
  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
🧬 Code graph analysis (1)
testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java (3)
testng-core/src/test/java/test/retryAnalyzer/issue3231/InvocationCountRetrySample.java (1)
  • InvocationCountRetrySample (9-27)
testng-core/src/test/java/test/retryAnalyzer/issue3231/MutationSample.java (1)
  • MutationSample (12-72)
testng-core/src/test/java/test/retryAnalyzer/issue3231/RetryLimitSample.java (1)
  • RetryLimitSample (9-30)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: 26, oracle, ubuntu, UTC, ru_RU
  • GitHub Check: 25, zulu, same hashcode, macos, UTC, tr_TR, stress JIT
  • GitHub Check: 17, zulu, ubuntu, America/New_York, fr_FR, stress JIT
  • GitHub Check: 17, microsoft, windows, UTC, ru_RU, stress JIT
  • GitHub Check: 21, oracle, windows, UTC, tr_TR
  • GitHub Check: 21, oracle, windows, UTC, ru_RU, stress JIT
  • GitHub Check: 11, microsoft, ubuntu, Pacific/Chatham, fr_FR
πŸ”‡ Additional comments (6)
testng-runner-api/src/main/java/org/testng/internal/TestResult.java (4)

39-39: LGTM!

The field initialization to -1 correctly aligns with the default return value specified in ITestResult.getParameterIndex(), serving as a clear sentinel for "unset."


357-362: LGTM! Guard logic correctly prevents conflicting reassignments.

The condition allows initial assignment and idempotent re-sets while throwing on conflicting values β€” this is the right protection for ensuring stable retry identification.


364-367: LGTM!

The @Override annotation ensures the interface contract is properly satisfied.


369-377: LGTM!

Adding @Override annotations to these existing interface implementations improves maintainability and ensures compile-time verification of the interface contracts.

Also applies to: 379-380, 456-458

testng-core/src/test/java/test/retryAnalyzer/RetryAnalyzerTest.java (2)

11-11: LGTM! Imports are appropriate.

The new imports support the invocation counting mechanism used in the tests below.

Also applies to: 15-16


46-48: LGTM! Sample class imports are correctly added.

These imports reference the test fixtures that exercise the issue #3231 fixes.

@baflQA
baflQA requested a review from krmahadevan January 6, 2026 18:26
@krmahadevan

Copy link
Copy Markdown
Member

@baflQA - Please check this patch. We could use something like this. We can also add a deprecation message to the setParameterIndex() in TestResult object so that we can get rid of that setter in a later release.

changes.patch

@baflQA

baflQA commented Jan 7, 2026

Copy link
Copy Markdown
Contributor Author

hey @krmahadevan - I've applied Your suggestions, tests are still passing ;) I think that we are approaching the happy ending :)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

πŸ€– Fix all issues with AI agents
In @testng-runner-api/src/main/java/org/testng/internal/TestResult.java:
- Around line 367-368: Mark the no-op setter setParameterIndex(int
parameterIndex) as deprecated: add the @Deprecated annotation to the method and
update its Javadoc with a @deprecated tag explaining that the method is a no-op
and will be removed in a future release (optionally mention the recommended
alternative or that callers should stop using it). Ensure the annotation and
Javadoc appear on the method signature for TestResult.setParameterIndex so tools
and IDEs show the deprecation to users.
πŸ“œ Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 849bb38 and 6763e50.

πŸ“’ Files selected for processing (4)
  • testng-core-api/src/main/java/org/testng/ITestResult.java
  • testng-core/src/main/java/org/testng/internal/IObject.java
  • testng-core/src/main/java/org/testng/internal/invokers/TestInvoker.java
  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
πŸ’€ Files with no reviewable changes (1)
  • testng-core/src/main/java/org/testng/internal/IObject.java
🚧 Files skipped from review as they are similar to previous changes (1)
  • testng-core-api/src/main/java/org/testng/ITestResult.java
🧰 Additional context used
🧠 Learnings (2)
πŸ“š Learning: 2024-10-09T09:04:04.750Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:11-16
Timestamp: 2024-10-09T09:04:04.750Z
Learning: The `retry(ITestResult result)` method must retain the `ITestResult` parameter because it implements the `IRetryAnalyzer` interface, and method signatures cannot be altered when implementing interfaces, even if the parameter is unused.

Applied to files:

  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
πŸ“š Learning: 2024-10-09T09:04:04.750Z
Learnt from: krmahadevan
Repo: testng-team/testng PR: 3181
File: testng-core/src/test/java/test/invocationcount/issue3180/RetryAnalyzer.java:6-9
Timestamp: 2024-10-09T09:04:04.750Z
Learning: In test code for the TestNG project, it's acceptable to hardcode parameters like `retryLimit` since they don't need to be configurable.

Applied to files:

  • testng-runner-api/src/main/java/org/testng/internal/TestResult.java
🧬 Code graph analysis (1)
testng-core/src/main/java/org/testng/internal/invokers/TestInvoker.java (1)
testng-runner-api/src/main/java/org/testng/internal/TestResult.java (1)
  • TestResult (25-501)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: 26, oracle, macos, Pacific/Chatham, fr_FR, stress JIT
  • GitHub Check: 21, temurin, windows, America/New_York, tr_TR
  • GitHub Check: 21, zulu, ubuntu, UTC, de_DE, stress JIT
  • GitHub Check: 11, liberica, windows, America/New_York, ru_RU
  • GitHub Check: 25, corretto, windows, Pacific/Chatham, de_DE
  • GitHub Check: 11, zulu, same hashcode, ubuntu, UTC, tr_TR, stress JIT
  • GitHub Check: 17, zulu, ubuntu, America/New_York, de_DE
πŸ”‡ Additional comments (6)
testng-runner-api/src/main/java/org/testng/internal/TestResult.java (5)

52-57: LGTM: Clean factory method for parameter-indexed results.

The factory method correctly initializes both parameters and parameter index in a single call, providing a cleaner API than separate setter calls.


88-89: LGTM: Consistent use of factory method.

Correctly uses the new factory to preserve both parameters and parameter index when creating a derived TestResult.


363-365: LGTM: Good encapsulation with private initialization.

The private visibility ensures parameter index can only be set through the factory method, preventing external modification.


369-382: LGTM: Proper @OverRide annotations.

The @OverRide annotations correctly indicate these methods implement the ITestResult interface.

Also applies to: 461-464


384-446: LGTM: @OverRide annotation added to interface implementation.

The @OverRide annotation correctly indicates this method implements ITestResult.getSkipCausedBy().

testng-core/src/main/java/org/testng/internal/invokers/TestInvoker.java (1)

595-596: LGTM: Correct usage of new factory method.

The change correctly uses TestResult.newTestResult() to create a TestResult with both parameter values and parameter index in a single call, replacing the previous pattern of creating an empty result and setting fields via setters.

baflQA added 3 commits January 7, 2026 21:47
Corrected several typos in comments and variable names within TestResult.java, such as 'atleast' to 'at least', 'interection' to 'intersection', and 'allfailures' to 'allFailures'. Also updated deprecated annotation to specify forRemoval=true for setParameterIndex.
Removed an extraneous asterisk from the Javadoc comment in the isNotRunning() method for improved readability.
@krmahadevan
krmahadevan merged commit be97321 into testng-team:master Jan 8, 2026
7 of 10 checks passed
@juherr

juherr commented Jan 8, 2026

Copy link
Copy Markdown
Member

@baflQA This process wasn’t short or simple. Thank you for your patience and for the quality of your contributions throughout the discussion. Looking forward to seeing more pull requests from you in the future.

@baflQA

baflQA commented Jan 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for Your and @krmahadevan support on this matter. This issue was bothering me for too long πŸ˜…

krmahadevan pushed a commit to krmahadevan/testng that referenced this pull request Jan 21, 2026
Following was done

- Add getParameterIndex() to ITestResult interface.
- Use parameterIndex in BaseTestMethod to identify test instances for RetryAnalyzer.
- This ensures a stable key even if parameters are mutated.
- Also fixes shared retry limit for rows with equal parameters.
- Add regression tests Issue3231Test and RetryLimitTest.

---------

Co-authored-by: Bartek Florczak <b.florczak@sap.com>
Co-authored-by: Julien Herr <julien@herr.fr>
@coderabbitai coderabbitai Bot mentioned this pull request Jan 21, 2026
@juherr juherr added this to the 7.12.0 milestone Sep 7, 2026 — with ChatGPT Codex Connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestNG retry is going into infinite loop when the data provider returned object is modified before failure

3 participants