Rerun Failed Tests in TestNG

In this post, we will learn to rerun failed test cases using TestNG. We will explore the two approaches to achieve this, namely – using the testng-failed.xml file and by implementing the testNG IRetryAnalyzer.

Rerun failed tests using testng-failed.xml

When to use?

Sometimes after some bug fixes, as test automation engineers, we are required to run only the failed tests reported by test automation suite. Running only the failed tests validate the bug fixes quickly.

How to achieve?

Running only the failed tests is fairly simple as TestNG provides inherent support for this. Whenever a test suite is run using the testng.xml file then after the test execution, a testng-failed.xml file gets created in the test-output folder. Later on, we can run this file just like we run the testng.xml file. As this file only keeps track of the failed tests, so running this file, runs the failed tests only.

Retry failed tests automatically using IRetryAnalyzer

When to use?

At times, the test execution report comes up with some failures that are not because of the issues in the application. The underlying cause of these issues might be related to the test environment setup or some occasional server issue. In order to make sure that the failure reported in the test report is genuine and not just one-off cases, we can retry running the failed test cases to eliminate false-negative test results in our test reports.

How to achieve?

For retrying the failure test runs automatically during the test run itself, we need to implement the IRetryAnalyzer interface provided by TestNG. The IRetryAnalyzer interface provides methods to control retrying the test runs. Here, we will override the retry() method of IRetryAnalyzer to make sure the test runs in case of failure with the specified number of retry limit. The comments in the code snippet make it self-explanatory.

Code Snippet

package com.artoftesting.test;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {

//Counter to keep track of retry attempts
int retryAttemptsCounter = 0;

//The max limit to retry running of failed test cases
//Set the value to the number of times we want to retry
int maxRetryLimit = 1;
//Method to attempt retries for failure tests
public boolean retry(ITestResult result) {
if (!result.isSuccess()) {
if(retryAttemptsCounter < maxRetryLimit){
retryAttemptsCounter++;
return true;
}
}
return false;
}
}

For the demo, I am creating a dummy test method below and intentionally failing it using the assert.fail() method. Here, we will set the @Test annotation’s retryAnalyzer attribute with RetryAnalyzer.class that we created above.

@Test(retryAnalyzer = RetryAnalyzer.class)
public void intentionallyFailingTest(){
System.out.println("Executing Test");
Assert.fail("Failing Test");
}

Test Output

===============================================
Test suite
Total tests run: 2, Failures: 1, Skips: 1
===============================================

Here, we can observe that the number of runs for the method are displayed as 2 with one failure and one skipped. The first run of the test method when failed will be marked as Skipped and then the test will run again due to the logic specified in the RetryAnalyzer.

Now, there is just one problem, we need to set the retryAnalyzer attribute in each of the @Test annotation. To deal with this we can implement IAnnotationTransformer interface. This interface provides the capability to alter the testNG annotation at runtime. So, we will create a class implementing the IAnnotationTransformer and make it set the RetryAnalyzer for @Test annotations.

package com.artoftesting.test;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.IRetryAnalyzer;
import org.testng.annotations.ITestAnnotation;
public class FailureRetryListener implements IAnnotationTransformer {

//Overriding the transform method to set the RetryAnalyzer
public void transform(ITestAnnotation testAnnotation, Class testClass,
Constructor testConstructor, Method testMethod) {
IRetryAnalyzer retry = testAnnotation.getRetryAnalyzer();
if (retry == null)
testAnnotation.setRetryAnalyzer(RetryAnalyzer.class);
}
}

Having created a listener, we can specify it in the testNG.xml file like this-

Now, we don’t have to set the retryAnalyzer in each @Test annotation. Having the listener specified in the testng.xml file will make it work for all the tests.

PS: If you want to add retry functionality to only a limited set of test methods than just set the @Test annotations with retryAnalyzer attribute with RetryAnalyzer.class, there is no need to implement IAnnotationTransformer and adding the listener in the testng.xml file.

Еще об этом:

TestNG Interview Questions КАК ЗАПУСТИТЬ НЕУДАЧНЫЕ ТЕСТ-КЕЙСЫ С ИСПОЛЬЗОВАНИЕМ TESTNG В SELENIUM WEBDRIVER 30 САМЫХ ПОПУЛЯРНЫХ ВОПРОСОВ И ОТВЕТОВ НА ИНТЕРВЬЮ Manual Testing Interview Questions and Answers What is Automation Testing? НЕПРЕРЫВНАЯ ИНТЕГРАЦИЯ SELENIUM С JENKINS [SELENIUM — MAVEN — GIT — JENKINS] — ПОШАГОВОЕ РУКОВОДСТВО What is a Test Case? How to write test cases? Unit Testing vs Integration Testing КАК СОЗДАТЬ XML-ФАЙЛ TESTNG И ВЫПОЛНИТЬ ФАЙЛ TESTNG.XML КАК СОЗДАТЬ ПРОЕКТ SELENIUM MAVEN В ECLIPSE IDE | РУКОВОДСТВО ПО СЕЛЕНУ