Finding test failures in Jenkins CI by replaying test program executionBill Burns | December 6, 2017
The continuous integration process includes routinely testing the application with the goal of finding bugs the moment any code changes are added to the repository. Inevitably, a test failure will occur and require an engineer’s time to analyze the failure, figure out how to address the failure, and make changes. There are several factors that often complicate the process for the engineer to diagnose and fix the failure:
• Intermittent failures: The test failure could be intermittent and hard to reproduce.
• Different development/test environments: The CI environment in which the test failed may not be accessible by the engineer and replicating the exact same environment to reproduce the failure may take effort and time.
What would be ideal is if the engineer could access a “recording” of the test program's execution at the time of the failure. The recording would be used to analyze the actual test run that produced the failure without the engineer replicating the test environment or requiring access to special machines, saving them time and allowing them to understand and fix the issue. Such a technology exists and is part of the CodeDynamics ReplayEngine execution recording engine.
My previous post on integrating memory leak detection into continuous integration gives a high-level overview of CodeDynamics program execution recording technology and how it is used in a CI environment to record test execution and provide recording files for later analysis. Here, we'll show how to integrate the technology directly into a Jenkins CI environment.
Jenkins and JUnit – An open source CI solution
There are several CI systems available for development teams, each with their own pros and cons. One of the popular ones is Jenkins. Jenkins is an open source project providing a self-contained automation server, used for automating different tasks including building a software application using a CI process. Plugins are available for Jenkins, extending its capabilities and functionality. One plugin for automated testing is JUnit. The JUnit plugin provides capabilities to consume XML test reports generated as part of building the product. The XML reports contain information about the tests run during a Jenkins job. JUnit consumes the XML test information and provides graphical visualizations, using the JUnit graph plugin, of the historical test results:
In addition to using JUnit, our solution leverages the JUnit Attachments plugin so that test program recording files are accessible as part of the test results presented in Jenkins.
To incorporate CodeDynamics recording results into Jenkins, we’ll leverage JUnit by generating JUnit XML consumable files with information about each test that generated a recording file.
Adding CodeDynamics test program recording to Jenkins
There are a few simple steps to record the execution of test programs under a Jenkins CI process.
Install JUnit Plugin for Jenkins
Typically, the JUnit plugin gets installed during the Jenkins installation. To double check, select Manage Jenkins from the dashboard, then select Manage Plugins. Click on the Installed tab and type “junit” in the filter box. JUnit Plugin should show up in the list. If it does not, click on the Available tab, search for “junit” again and install the plugin.
Install JUnit Attachments Plugin for Jenkins
Using the same approach, first select Manage Jenkins from the dashboard, then select Manage Plugins. Click on the Available tab and type “junit attachments” in the filter box. The JUnit Attachments Plugin should show up in the list. Install it.
Run tests under the control of CodeDynamics recording engine in Jenkins
With Jenkins and the needed JUnit plugins prepared, it’s time to record the execution of the test applications.
For this example, we will run a test application that crashes, see the Further Reading section below for links to the sources to tx_simple_crash. The tx_simple_crash application can be built as part of the Jenkins job or by executing the following command from the terminal and making the final executable accessible to our Jenkins test project.
g++ -g tx_simple_crash.cxx -o tx_simple_crash
To configure a Jenkins test project, first start from your Jenkins dashboard and click on New Item.
For this session, we create a Freestyle project to record the execution of the test application. Enter an item name for the project, click on Freestyle project, and then click the OK button.
Jenkins creates the new Freestyle project and presents a page with various options. For this example. we’re only concerned with the Build section. Click on the Build tab at the top of the page and then select Execute shell from the Add build step menu.
Jenkins will present the following form for providing a command to execute within a shell. It is here that we are going to run our tx_simple_crash test application under the control of the CodeDynamics’ execution recording engine, called ReplayEngine, using its scripting interface, tvscript.
Enter the following command into the Command field for Execute shell.
/opt/toolworks/codedynamics/bin/tvscript -replay \ -event_action "error=>save_replay_recording_file" \ tx_simple_crash
Note: The path to tvscript may need to be adjusted depending on where CodeDynamics is installed.
In this scenario, we are running the target application under the control of tvscript directly within Jenkins. Depending on the test harness you run your test applications under, you will apply the same technique.
The CodeDynamics’ script, tvscript, provides a variety of facilities to run an application in a non-interactive way. Its whole goal is to monitor the execution of the application, pick up an error, and provide mechanisms for analyzing and debugging an application. Tvscript provides the ability to record the execution of a target application and save the recording out to a file. Developers can load the file into CodeDynamics for replay and debugging.
To turn on recording for a target application, you provide the -replay flag to tvscript but tvscript also needs to know when to save the recording file. In this case, we want to save the recording file on application crash. This is accomplished by providing an event/action pair instruction to tvscript using -event_action “error=>save_replay_recording_file” as a command line argument. The final argument to tvscript is the name of the target application run under the control of tvscript. There are numerous other events and actions that tvscript can perform. Issue tvscript -help for a quick summary or check out tvscript’s online documentation and check out section “Other ways to use tvscript to generate recordings files” below for some more examples.
The job is now configured to run the test application under tvscript and save an execution history recording file if the target application crashes. When Jenkins runs our job, tvscript will generate a human-readable log file with textual information about any problems. The final step is to detect that a recording file was generated, mine information out of the tvscript log files, and prepare the information to be displayed in Jenkins through the JUnit plugin.
Converting tvscript .log files into JUnit XML files
Currently, tvscript generates text log file which contains human readable information about any errors generated by the application or if any recording files were generated by the CodeDynamics ReplayEngine. An example log file is as follows:
******************************************************************************* * TotalView Debugger Script Log File * * Date: 08-23-2017_15:11:58 * Target: tx_simple_crash * Event/Action Directives: * error => save_replay_recording_file ******************************************************************************* Running target tx_simple_crash !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! Process Error Notification ! ! Process: ! tx_simple_crash (Debugger Process ID: 1, System ID: 86332) ! Thread: ! Debugger ID: 1.1, System ID: 86332 ! Error: ! Segmentation violation ! Error Location: ! > 0 godump PC=0x0040061a, FP=0x7ffd5ddab680 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash.cxx#17] ! 1 main PC=0x00400659, FP=0x7ffd5ddab6a0 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash.cxx#24] ! 2 __libc_start_main PC=0x7f15b3705f43, FP=0x7ffd5ddab760 [/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so] ! 3 _start PC=0x004004f4, FP=0x7ffd5ddab768 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ** Performing action save_replay_recording_file for event error ** 08-23-2017 15:11:58 - Saving replay recording file to file tx_simple_crash-86332_08-23-2017_15:11:58.1.recording
To bring this information into Jenkins, we are leverage JUnit’s capabilities to read test result information from XML files.
Using a handy Python script (download here, you'll need to remove the .txt extension), we are going to convert tvscript log file information into JUnit’s XML file format. When run, the script takes the JUnit XML filename to produce and one or more tvscript generated log files.
python tvscript2junit.py recordings.xml *.log
For a single log file, the script will produce an XML file like the following:
<!--?xml version="1.0" encoding="UTF-8" ?--> <testsuites name="Replay Recordings" tests="1" failures="1"> <testsuite name="Replay Recordings" tests="1" failures="1"> <testcase classname="recordings" name="tx_simple_crash"> <error message="tx_simple_crash exited due to an error."> <![CDATA[ ! Process Error Notification ! ! Process: ! tx_simple_crash (Debugger Process ID: 1, System ID: 86332) ! Thread: ! Debugger ID: 1.1, System ID: 86332 ! Error: ! Segmentation violation ! Error Location: ! > 0 godump PC=0x0040061a, FP=0x7ffd5ddab680 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash.cxx#17] ! 1 main PC=0x00400659, FP=0x7ffd5ddab6a0 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash.cxx#24] ! 2 __libc_start_main PC=0x7f15b3705f43, FP=0x7ffd5ddab760 [/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so] ! 3 _start PC=0x004004f4, FP=0x7ffd5ddab768 [/mnt/development/bburns/.jenkins/workspace/record-test/tx_simple_crash] ]]> </error> </testcase> </testsuite> </testsuites>
Finally, there’s one more step to perform and that is to attach the recordings file to the Jenkins test results.
Attaching recording files to JUnit test results using JUnit Attachments
In addition to the generated log files, if the program crashed, tvscript generates execution history recordings. Downloading the recording file directly from Jenkins test pages enables developers to begin analysis of what went wrong. To attach the recordings files to the Jenkins test results we’ll leverage the JUnits Attachment plugin.
JUnit Attachments provides two mechanisms for attaching files:
1) Placing files in a directory named the same as the test class
2) Printing out the file name in a format that Jenkins will understand
We’ll follow the first pattern.
When Jenkins runs our job, all tvscript log files and recordings files are placed in a Jenkins workspace directory. When we analyze the tvscript log files to produce the JUnit XML file, it too is placed in the workspace directory. Using the JUnit Attachments mechanism of placing all files in a directory named the same as the test class, we can attach any recordings files to the JUnit test results by placing them in the test class directory. The actual process of doing this is simple and occurs when running the tvscript2junit.py script. First, using a test class named “recordings”, create a directory named “recordings” in the Jenkins workspace directory. Next, for each tvscript log file that included a recordings file, create a link to the file from the recordings directory. The generated JUnit XML file also needs to mark each test as part of the “recordings” test class.
Generate the JUnit XML file and create the links to recordings files from within a “recordings” test class directory, by adding another Execute shell build step to the project and enter the tvscript2junit.py command. Place the tvscript2junit.py script in the Jenkins workspace directory. You may need to adjust paths based on where your tvscript2junit.py script is located if you put it in a different location.
Our project is ready to run. It will generate a recordings file if the test crashes and converts the tvscript generated log files into JUnit consumable XML files. The final step before we run our project is to tell Jenkins where to fetch the JUnit XML files and publish a JUnit test results report.
Publishing JUnit test results report with recordings information
In order for JUnit to generate test reports, it needs know where to put the XML files with test result data. To do this, click on Add post-build action at the bottom of our project configuration and then select Publish JUnit test result report. As an aside, the XML files generated by tvscript2junit.py are compatible with the xUnit test reporting plugin.
The following form is displayed. This form allows you to specify where to pick up JUnit XML files. We can simply enter tvscript.xml in the Test report XMLs field since we generate the tvscript.xml files into the Jenkins workspace area.
Before saving the project, click the “Add” button for Additional test report features and select Publish test attachments. This engages the JUnits Attachment plugin to scan the JUnit test results and pick the recordings and attach them to the test report.
Finally, click Save to save our project.
Generating execution history recordings and accessing them in Jenkins
With our project configured, click Build Now from the project dashboard.
Jenkins runs our project and show the most recent build in the Build History on the dashboard.
The build is marked as yellow, indicating there is a problem. Let’s explore the test results.
Exploring recordings test results in Jenkins
Clicking on the build #1 link in the Build History section brings up a summary of the status.
There is one test failure in the tx_simple_crash test application. Clicking on the link gives details about the failure. This information was captured from the tvscript log file.
The Error Message reports the test failed due to an error. The “Stacktrace” area provides the contents of the log file generated by tvscript, including details about the error.
The recordings file is not available at this location since it is an attachment and the JUnit Attachments plugin associates all attachments at the class level. To find the attachments click on the “recordings” link in the bread crumb links.
Jenkins will display a page with all the tests that generated recordings files attachments.
The developer can download the recording file and load it into CodeDynamics. Within CodeDynamics, they can conduct debugging and move forwards and backwards through execution history to determine the cause of the crash.
To learn more about CodeDynamics reverse debugging capabilities check out the Reverse debugging on CodeDynamics with ReplayEngine video.
Other ways to use tvscript to generate recordings files
The example in this article generated an execution recording file if the test program crashes. It’s not always the case that tests result in a crash but instead, simply fail with erroneous results. In these cases, tvscript can be instructed to generate recording files if a certain location in code indicating the failure is hit, just before the program is due to exit or based on custom conditions that you code into your own tvscript action handler. Here are a few examples:
Save a recording file just before the program exits:
tvscript -replay \ -event_action “stopped_at_end=>save_replay_recording_file” \ myTestProgram
Save a recording file if a line 85 of file myfile.cxx is hit:
tvscript -replay \ -create_actionpoint “myfile.cxx#85=>save_replay_recording_file” \ myTestProgram
Read more about how to fully utilize in the CodeDynamics reference guide.
Incorporating CodeDynamics test program recording into your Continuous Integration (CI) environment provides many benefits including giving developers the ability to diagnose test failures by loading recording files into CodeDynamics to examine the failure that occurred. No additional effort is needed to reproduce the test environment and the circumstances that caused the test failure. The tvscript2junit.py script makes it easy to convert tvscript log files into XML files that incorporate into the Jenkins CI process. The XML files are also compatible with the xUnit XML file format, allowing them inclusion into xUnit testing results too.
Although this article focused on integration with Jenkins, integration with other CI systems that understand JUnit or xUnit XML files follow a similar approach.
For more information other CI solutions available from Rogue Wave, including leak detection and static code analysis into your CI system using our Klocwork product, visit roguewave.com.
• Blog post on how to integrate leak detection into any CI tool
• tvscript2junit.py (remove the .txt extension)
• tx_simple_crash.cxx (remove the .txt extension)
• How Jenkins CI parses and displays JUnit output
• JUnit Plugin
• JUnit Attachments Plugin
• xUnit Plugin