Displaying .NET Core XUnit Test Results( run from Cake) in Jenkins/SonarQube

This is a bit of a brain dump.  I’ve had an entire OneNote section called “XUnit Nightmare” for about six months and finally got around to getting it into some kind of order to share with some of the newer guys in the team.

In short, we were have XUnit tests which are run from a Cake script, and we want to display the results in Jenkins and/or Sonarqube.

TL;DR; .NET Core outputs a .TRX file.  Jenkins (and SonarQube) can’t seem to handle it.  The content of the TRX file is XML, so we use XSLT to convert to the JUnit format, which is better supported in these tools.

This was more difficult than it should have been.  We can execute the tests form the command like using:

dotnet test

There are other parameters you can provide, but if you’re in a folder containing a project/solution and there are tests, it will run them all.  Run ‘dotnet test –help’ to see all the options.

We’re executing the tests using Cake using the DotNetCoreTest command, which just calls out to the dotnet test command.  The Cake documentation is actually really good – https://cakebuild.net/api/Cake.Incubator.DotNetCoreTestExtensions/DotNetCoreTestExtensions/BFD20FC8.

TRX Test Output

Looking at dotnet test –help you’ll see the option to output to a file is

 --logger "trx;LogFileName=MyTestResults.trx"

Breaking Change in SDK 2.2.202

Newer SDK versions have caused the ‘LogFileName’ parameter to effectively became ‘LogFilePrefix’ – a change made in these version adds a timestamp before the ‘.trx’ in the file name.  We had to make changes to the Cake script to support this.  This issue is detailed here:
https://github.com/Microsoft/vstest/issues/1951.

This change appeared in the first versions of the SDKs to support Visual Studio 2019, so for 2.1 that was 2.1.604 and for 2.2 it was 2.2.202.  Be careful with these SDKs, because with the support for a new version of Visual Studio being added – it looks like a minor version upgrade to the SDK but it can be a major version upgrade for some underlying tools (which may contain breaking changes).

https://dotnet.microsoft.com/download/dotnet-core/2.1

https://dotnet.microsoft.com/download/dotnet-core/2.2

TRX Not Supported by Jenkins

Unfortunately, this file format is not supported by Jenkins, and getting dotnet test to output in a different format wasn’t happening – explained below.  We needed to convert this into a format which Jenkins could understand.

Getting the XSLT

If you open a TRX file with a text editor, you’ll see the content is just XML.  A bit of googling led to this XSLT which we can use to transform it into the format output by JUnit:

https://github.com/ekyoung/personal-site-dotnet/blob/master/trx-to-junit.xslt

Updating the XSLT for .NET Core

We made a change to the className selectors on lines 52/53 and 115/116 to only include ‘.Tests’ namespaces.  I’m not sure whether perhaps the TRX output has changed slightly since the XSLT was written or maybe it was something specific to the Jenkins plugin for JUnit results that needed that change.

Using it in Cake

Using this XSLT we were able to transform the TRX output into JUnit format.  We did this with the Cake ‘XmlTransform’ command…

https://cakebuild.net/api/Cake.Common.Xml/XmlTransformationAlias/7912BB19

Displaying in Jenkins

Once we had the results in JUnit format, I believe this was something Jenkins could then display relatively easily (or at least, the DevOps guys made it look easy).

 

Problems Along The Way

For XUnit tests, use the ‘DotNetCoreTest’ Cake command, rather than ‘XUnit’ or ‘XUnit2’

The cake commands just call out to the underlying .NET tools, so ‘dotnet test’ and ‘dotnet xunit’ and in the past using the xunit commands/tools has given more specific options – we were looking at this to change the output format.

From the information online, this looked quite easy – the ‘xunit’ command has a ‘/nunit’ switch to output in NUnit format.  The problem is that in newer version we must invoke it via ‘dotnet test’:

“For 2.x, the only supported way to run .NET Core tests is with the xunit.runner.visualstudio package (and either Test Explorer inside Visual Studio and/or the dotnet test command line)”.

https://github.com/xunit/xunit/issues/1358

And because we must invoke xunit indirectly we can’t pass through parameters like ‘/nunit’ which ‘dotnet test’ does not support.

https://github.com/xunit/xunit/issues/1839

Using TRX Files in SonarQube

According to this page, SonarQube should be able to make use of TRX files
https://docs.sonarqube.org/pages/viewpage.action?pageId=6389772

Our experience suggested otherwise.  Perhaps the TRX files from .NET Core were in a slightly newer format.

The main problem we were having is that it would import the number of tests, but not list the test details.  These posts relate to the problems:

https://github.com/SonarSource/sonar-dotnet/issues/492#issuecomment-320704456

https://github.com/SonarSource/sonar-dotnet/issues/886

https://jira.sonarsource.com/browse/SONARCS-657

https://andrastornai.com/2016/07/31/sonarqube-and-csharp-drill-down-results-of-test-cases/

Converting TRX to SonarQube Format using XSLT

https://github.com/awesley/trxToSonarQubeXslt/blob/master/trxToSonarQubeGenericTestData.xsl

We also had the problem where we couldn’t import the test results into SonarQube and found a similar XSLT.  Unfortunately, we found it difficult to use because it uses XSLT 3.0 which is not very widely supported supports…

https://github.com/maxtoroq/dotnet-xml#xpath-xquery-and-xslt

Neither .NET (https://blogs.msdn.microsoft.com/dareobasanjo/2004/05/13/why-you-wont-see-xslt-2-0-or-xpath-2-0-in-the-next-version-of-the-net-framework/) or Linux tools like XSLTProc support it.  Tools with XSLT 3.0 support are a proprietary tool called Exselt (who’s website, http://exselt.net/, mostly times out) and Saxon (http://saxon.sourceforge.net/).

Downgrading the XSLT

I had a little look into reworking the XSLT into a format that .NET/Cake/XSLTProc could understand – an earlier XSLT version.  The issue was the use of the ‘<xsl:for-each-group’ element, which is not supported in earlier versions.

Muenchian grouping

http://www.jenitennison.com/xslt/grouping/muenchian.html
https://en.wikipedia.org/wiki/XSLT/Muenchian_grouping

This looks to be the method to factor out a ‘for-each-group’, but seeing it wasn’t a 5 minute job, I didn’t go much further with that.

Saxon

Saxon (http://saxon.sourceforge.net/) looked to be a good tool for the job.  It was mentioned on a GitHub issue reporting the SonarQube issues  (https://github.com/SonarSource/sonar-dotnet/issues/886)  but we had some problems getting it to run.  Using the free Saxon HE edition, there are Java and .NET versions available.  We did not have Java setup in the Linux build environment, and the .NET version is .NET Framework.  We attempted to run this on Linux with Mono but had further problems.  When we eventually did some testing on Windows we were able to get it to convert but SonarQube still wouldn’t import it.

Saxon .NET Core

https://saxonica.plan.io/issues/2871

According to this post, it’s unlikely we’ll see a version of Saxon based on .NET Core, because the .NET Framework version is a port of the Java version and the tool used for that port, IKVM, does not support .NET Core.  The person responsible for IKVM posted “I don’t know if it is possible, but I do know it will be a lot of work and it isn’t going to happen anytime soon.”  – https://sourceforge.net/p/ikvm/mailman/message/35274049/

 

Writing our own tool to do this

I’ve thought about this a bit.  It wouldn’t be that difficult – the formats are all published (details below).  We would use .NET Core, so it would be cross platform, and it would be entirely under our control.  There is the cost of having to maintain this tool – when file formats change over time for example, but I would say it’s better to that than to be relying on scripts and XSLTs downloaded from the internet which nobody within the team really understands.

https://docs.sonarqube.org/latest/analysis/generic-test/

 

 

 

 

Advertisement

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s