Repository navigation
Closed
Labels
area: listenersListeners, method interceptors, and invocation callbacksListeners, method interceptors, and invocation callbacks
Milestone
Description
TestNG Version
Note: only the latest version is supported
7.6.1
Expected behavior
16:02:12.583 [main] INFO TestNgDemoListener - after invocation executed, method:beforeClass,testResultStatus:FAILURE/2
java.lang.ArithmeticException: / by zero
16:02:12.631 [main] INFO TestNgDemoListener - before configuration executed, method:beforeMethod,testResultStatus:**SKIP/3**
16:02:12.631 [main] INFO TestNgDemoListener - before invocation executed, method:beforeMethod,testResultStatus:**SKIP/3**
16:02:12.632 [main] INFO TestNgDemoListener - after invocation executed, method:beforeMethod,testResultStatus:SKIP/3
16:02:12.633 [main] INFO TestNgDemoListener - Skip Config Method,beforeMethod,testResultStatus:SKIP/3
14:03:16.659 [main] INFO TestNgDemoListener - onTestStart executed, method:issueTest,testResultStatus:STARTED/16
14:03:16.661 [main] INFO TestNgDemoListener - onTestSkipped executed, method:issueTest,testResultStatus:SKIP/3
14:03:16.660 [main] INFO TestNgDemoListener - before invocation executed, method:issueTest,testResultStatus:SKIP/3
14:03:16.660 [main] INFO TestNgDemoListener - after invocation executed, method:issueTest,testResultStatus:SKIP/3
Actual behavior
16:02:12.583 [main] INFO TestNgDemoListener - after invocation executed, method:beforeClass,testResultStatus:FAILURE/2
java.lang.ArithmeticException: / by zero
16:02:12.631 [main] INFO TestNgDemoListener - before configuration executed, method:beforeMethod,testResultStatus:**STARTED/16**
16:02:12.631 [main] INFO TestNgDemoListener - before invocation executed, method:beforeMethod,testResultStatus:**STARTED/16**
16:02:12.632 [main] INFO TestNgDemoListener - after invocation executed, method:beforeMethod,testResultStatus:SKIP/3
16:02:12.633 [main] INFO TestNgDemoListener - Skip Config Method,beforeMethod,testResultStatus:SKIP/3
14:03:16.659 [main] INFO TestNgDemoListener - onTestStart executed, method:issueTest,testResultStatus:STARTED/16
14:03:16.661 [main] INFO TestNgDemoListener - onTestSkipped executed, method:issueTest,testResultStatus:SKIP/3
14:03:16.660 [main] INFO TestNgDemoListener - before invocation executed, method:issueTest,testResultStatus:**SKIP/3**
14:03:16.660 [main] INFO TestNgDemoListener - after invocation executed, method:issueTest,testResultStatus:SKIP/3
Is the issue reproducible on runner?
- Shell
- Maven
- Gradle
- Ant
- Eclipse
- IntelliJ
- NetBeans
Test case sample
Please, share the test case (as small as possible) which shows the issue
Simply use listener:
public class TestNgDemoListener implements IInvokedMethodListener, IConfigurationListener, ITestListener
import org.slf4j.LoggerFactory;
import org.testng.IConfigurationListener;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
public class TestNgDemoListener implements IInvokedMethodListener, IConfigurationListener, ITestListener {
org.slf4j.Logger logger = LoggerFactory.getLogger(TestNgDemoListener.class);
@Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
logger.info(
"before invocation executed, method:" + testResult.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(testResult.getStatus()));
}
@Override
public void beforeConfiguration(ITestResult testResult) {
logger.info(
"before configuration executed, method:" + testResult.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(testResult.getStatus()));
}
@Override
public void onConfigurationSkip(ITestResult testResult, ITestNGMethod method) {
logger.info("Skip Config Method," + testResult.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(testResult.getStatus()));
}
@Override
public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
logger.info(
"after invocation executed, method:" + testResult.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(testResult.getStatus()));
}
@Override
public void onTestStart(ITestResult result) {
logger.info(
"onTestStart executed, method:" + result.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(result.getStatus()));
}
@Override
public void onTestSkipped(ITestResult result) {
logger.info(
"onTestSkipped executed, method:" + result.getMethod().getMethodName() + ",testResultStatus:"
+ getStatusString(result.getStatus()));
}
public static String getStatusString(int status) {
return switch (status) {
case -1 -> "CREATED/" + status;
case 1 -> "SUCCESS/" + status;
case 2 -> "FAILURE/" + status;
case 3 -> "SKIP/" + status;
case 4 -> "SUCCESS_PERCENTAGE_FAILURE/" + status;
case 16 -> "STARTED/" + status;
default -> "Others/" + status;
};
}
}import org.slf4j.LoggerFactory;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeGroups;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners({ TestNgDemoListener.class})
public class TestNgDemoTest {
org.slf4j.Logger logger = LoggerFactory.getLogger(TestNgDemoTest.class);
@BeforeClass
public void beforeClass() {
logger.info("beforeClass");
int i = 5/0; // throw an exception
}
@BeforeMethod
public void beforeMethod() {
logger.info("beforeMethod");
}
@BeforeTest
public void beforeTest() {
logger.info("beforeTest");
}
@BeforeSuite
public void beforeSuite() {
logger.info("beforeSuite");
}
@BeforeGroups
public void beforeGroups(){
logger.info("beforeGroups");
}
@Test
public void issueTest() {
logger.info("Test BeforeConfiguration behavior");
}
}Contribution guidelines
Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.
Description
After one of our test project uplift TestNG from 7.5 to 7.6 version, there are some unexpected behavior from before. the related issue is:
beforeConfiguration() listener method should be invoked for skipped configurations as well · Issue #2729 · cbeust/testng · GitHub
Metadata
Metadata
Assignees
Labels
area: listenersListeners, method interceptors, and invocation callbacksListeners, method interceptors, and invocation callbacks
Activity
krmahadevan commentedon Mar 9, 2023
@bobshie -Please include a sample (preferably a sample project) that can be used to reproduce the problem
bobshie commentedon Mar 9, 2023
@krmahadevan , Thanks, I add an example and there is conflict with:
it should be the same status in before invocation of 'beforeMethod' and 'issueTest'
Hkh9966 commentedon Mar 9, 2023
Hi @krmahadevan , in order to simply reproduce this problem, I did some simple code and drew a flow of the issue that is being discussed now:
If there are three methods:
And a customized listener:
The execution process is performed as shown in the figure:

In my opinion,since a config method(just like beforeMethod in this demo) has been determined to be skipped, the status obtained in beforeConfiguration() and beforeInvocation() for config method should be skip. In this way, config method and @test method are consistent.
And I also found a phenomenon in which @OnTestStart and beforeInvocation for @test method also behave inconsistently. I’m not sure if TestNg’s definition of these two methods is different.
As you know, since this issue was solved: #2729. If there is a situation in the demo, beforeConfiguration() will also be invoked, and we will get the status as STARTED, which will add a lot of changes for our project.
krmahadevan commentedon Mar 9, 2023
@Hkh9966 - Since you have spent time on analysing this, would you be willing to help fix it by raising a PR?
bobshie commentedon Mar 10, 2023
@krmahadevan , I'll try to do it.
Can we fix it as:
the expected status in beforeConfiguration & beforeInvocation should be 'SKIP' when beforeMethod is 'SKIP'.
and the expected status in onTestStart & beforeInvocation should be 'SKIP' when test method is 'SKIP'.
krmahadevan commentedon Mar 10, 2023
@bobshie sure. It also goes without saying that this fix should not cause regression and all existing tests work fine.
@juherr what do you think ?
Hkh9966 commentedon Mar 10, 2023
If we can change it to:
"The status in beforeConfiguration & beforeInvocation should be 'SKIP' when beforeMethod is 'SKIP'."
It is enough to solve our problems.
For "onTestStart & beforeInvocation should be 'SKIP' when test method is 'SKIP'", it may be that the changes will be large. And I'm not sure if this change is justified.
krmahadevan commentedon Mar 10, 2023
@Hkh9966
In that case, can we please file do the following ?
bobshie commentedon Mar 10, 2023
@krmahadevan , I think there is clear description in this thread, we only need to change the first one:
The status in beforeConfiguration & beforeInvocation should be 'SKIP' when beforeMethod is 'SKIP'."
about the next issue, testng has onTestStart() and onTestSkip() when test method is skipped:
so we can't change
onTestStart(STARTED/16)toonTestStart(SKIP/3). it's also very confusing.I can just fix the first one if it's agreed.
Hkh9966 commentedon Mar 10, 2023
@krmahadevan I agree with you.
change before config/invocation status 'SKIP' testng-team#2880
Skip config listener calls for skipped configs