Format test reports

Results on the Tests tab are parsed from test reports specified in the Report Paths setting in Run and Test steps. Test reports must be in JUnit XML format to appear on the Tests tab, because Harness parses test reports that are in JUnit XML format only.

For optimal rendering in the Harness UI, there is a limit of 8,000 characters per field. If a field in your XML file contains more than 8,000 characters, the output might render incorrectly on the Tests tab.

JUnit XML format resources

Use these resources to learn about JUnit XML formatting.

• Common JUnit XML format and examples from Testmo
• Examples of JUnit XML format mapping from IBM
• Apache Ant JUnit schema

Tools with built-in JUnit XML output

Here are some Harness YAML examples for test tools that produce JUnit XML output by default.

C, C++

CTest Output JUnit

You can use the --output-junit command with CTest.

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Sh
      command: |-
        mkdir build
        cmake -S . -B build
        ctest --test-dir build --output-junit out.xml
      reports:
        type: JUnit
        spec:
          paths:
            - /harness/build/out.xml

Java - Gradle

Gradle

Run step:

              - step:
                  type: Run
                  ...
                  spec:
                    shell: Sh
                    command: |-
                     test --tests
                    ...
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - "/harness/results.xml"
                      type: JUnit

Test step:

              - step:
                  type: Test
                  name: Intelligent Tests
                  identifier: test
                  spec:
                    command: test --tests
                    shell: Sh
                    ...
                    reports:
                      - "/harness/results.xml"

PHP

PHPUnit

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Sh
      command: |-
        mkdir -p /harness/phpunit
        phpunit --log-junit /harness/phpunit/junit.xml tests
      reports:
        type: JUnit
        spec:
          paths:
            - /harness/phpunit/junit.xml

Python

For Python, use pytest or unittest. You can also use pytest to run unittest.

Pytest in a Run step

This example runs pytest in a Run step.

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Sh
      command: |-
        . venv/bin/activate
        mkdir /harness/test-results
        pytest --junitxml=harness/test-results/junit.xml
      reports:
        type: JUnit
        spec:
          paths:
            - /harness/test-results/junit.xml

info

If you use test splitting with pytest in a Run step, you must set junit_family=xunit1 in your code repo's pytest.ini file or include -o junit_family="xunit1" in the step's command.

Pytest with Test Intelligence (Test step)

This example runs pytest with Test Intelligence.

              - step:
                  type: Test
                  name: Intelligent Tests
                  identifier: test
                  spec:
                    command: |-
                      python3 -m venv .venv
                      . .venv/bin/activate

                      python3 -m pip install -r requirements/test.txt
                      python3 -m pip install -e .

                      pytest --junitxml=out_report.xml
                    shell: Python
                    ...
                    reports:
                      - out_report.xml*

Ruby - Cucumber

Cucumber

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Sh
      command: |-
        bundle check || bundle install
        mkdir -p /harness/cucumber
        bundle exec cucumber --format junit --out /harness/cucumber/junit.xml
      reports:
        type: JUnit
        spec:
          paths:
            - /harness/cucumber/junit.xml

JUnit converters, formatters, and plugins

If your test tool doesn't automatically produce test results in JUnit XML format, there are JUnit converters, formatters, and plugins available for all major languages. Some examples of conversion tools and corresponding Harness YAML are provided below.

C# - .NET Core, NUnit

• NUnit to JUnit
• .NET trx2JUnit

Example: NUnit to JUnit

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Powershell
      command: |-
        cd dotnet-agent/TestProject1
        wget -UseBasicParsing https://dot.net/v1/dotnet-install.ps1 -o dotnet-install.ps1
        .\dotnet-install.ps1
        dotnet build

        wget https://raw.githubusercontent.com/nunit/nunit-transforms/master/nunit3-junit/nunit3-junit.xslt -o nunit3-junit.xslt

        "C:/Program Files (x86)/NUnit.org/nunit-console/nunit3-console.exe" dotnet-agent/TestProject1/bin/Debug/net48/TestProject1.dll --result="UnitTestResults.xml;transform=nunit3-junit.xslt"
      reports:
        type: JUnit
        spec:
          paths:
            - UnitTestResults.xml

Clojure

• Kaocha JUnit Plugin
• Clojure.test JUnit Plugin

Go

Go Junit Report

You can use the go-junit-report tool.

- step:
    type: Run
    identifier: test
    name: Test
    spec:
      shell: Sh
      command: |-
        go install github.com/jstemmer/go-junit-report/v2@latest
        go test -v ./... | tee report.out
        cat report.out | $HOME/go/bin/go-junit-report -set-exit-code > report.xml
      reports:
        type: JUnit
        spec:
          paths:
            - report.xml

Java - Maven

Maven Surefire Plugin

This example uses the Maven Surefire Plugin.

Run step:

              - step:
                  type: Run
                  ...
                  spec:
                    shell: Sh
                    command: |-
                     test -Dmaven.test.failure.ignore=true -DfailIfNoTests=false
                    ...
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - "target/surefire-reports/*.xml"
                      type: JUnit

Test step:

              - step:
                  type: Test
                  name: Intelligent Tests
                  identifier: test
                  spec:
                    command: test -Dmaven.test.failure.ignore=true -DfailIfNoTests=false
                    shell: Sh
                    ...
                    reports:
                      - "target/surefire-reports/*.xml"

JavaScript

ESLint

ESLint JUnit Formatter

- step:
    type: Run
    name: Run ESLint Tests
    identifier: run_eslint_tests
    spec:
      shell: Sh
      command: |
        mkdir -p /harness/reports
        eslint ./src/ --format junit --output-file /harness/reports/eslint.xml
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/reports/eslint.xml"

Jest

Jest JUnit Reporter

- step:
    type: Run
    name: Run Jest Tests
    identifier: run_jest_tests
    spec:
      shell: Sh
      command: |
        yarn add --dev jest-junit
        jest --ci --runInBand --reporters=default --reporters=jest-junit
    envVariables:
      JEST_JUNIT_OUTPUT_DIR: "/harness/reports"
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/reports/*.xml"

Karma

Karma JUnit Reporter

- step:
    type: Run
    name: Run Karma Tests
    identifier: run_karma_tests
    spec:
      shell: Sh
      command: |
        npm install
        mkdir /harness/junit
        karma start ./karma.conf.js
    envVariables:
      JUNIT_REPORT_PATH: /harness/junit/
      JUNIT_REPORT_NAME: test-results.xml
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/junit/test-results.xml"

Mocha

Mocha JUnit Reporter

- step:
    type: Run
    name: Run Mocha Tests
    identifier: run_mocha_tests
    spec:
      shell: Sh
      command: |
        npm install
        mkdir /harness/junit
        mocha test --reporter mocha-junit-reporter --reporter-options mochaFile=./path_to_your/file.xml
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/junit/test-results.xml"

Ruby - Minitest, RSpec

Minitest

Add the Minitest Junit Formatter to your Gemfile.

- step:
    type: Run
    name: Run Ruby Tests
    identifier: run_ruby_tests
    spec:
      shell: Sh
      command: |
        bundle check || bundle install
        bundle exec rake test --junit
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/report.xml"

RSpec

Add the RSpec JUnit formatter to your Gemfile.

- step:
    type: Run
    name: Run RSpec Tests
    identifier: run_rspec_tests
    spec:
      shell: Sh
      command: |
        bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
        mkdir /harness/rspec
        bundle exec rspec --format progress --format RspecJunitFormatter -o /harness/rspec/rspec.xml
    reports:
      type: JUnit
      spec:
        paths:
          - "/harness/rspec/rspec.xml"

Code coverage reports and test report artifacts

For information about code coverage reports and publishing report URLs to the Artifacts tab, go to Code Coverage.

Troubleshoot test reports

Go to the CI Knowledge Base for questions and issues related to test reports in Harness CI, including:

• Test suites incorrectly parsed
• Test reports missing
• Test report truncated
• Multiple test report paths
• Test Intelligence call graph is empty