Coverage badge in Gitlab CI with Python coverage always unknown

Question

I am trying to show a coverage badge for a Python project in a private Gitlab CE installation (v11.8.6), using coverage.py for Python. However, the badge always says unknown.

This is the relevant job in my .gitlab-ci.yaml file:

coverage:
    stage: test
    before_script:
        - pip3.6 install coverage
        - mkdir -p public
    script:
        - coverage run --source=my_service setup.py test
        - coverage report | tee public/coverage.txt
    artifacts:
        paths:
            - public/coverage.txt
    coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'

I expected the badge to show the actual coverage at this URL, so this is what I have entered in the project settings under General/Badges:

http://<privategitlaburl>/%{project_path}/badges/%{default_branch}/coverage.svg?job=coverage

I read these instructions using Gitlab pages. However, I do not want to use pages just for this purpose, and I am dealing with a Python project.

According to the example in the CI/CD settings, and in this post, the regex in the coverage entry should work. which I could confirm by trying it locally:

$ grep -P "TOTAL\s+\d+\s+\d+\s+(\d+%)" public/coverage.txt
TOTAL                                           289     53    82%

I also tried the same regex in the field Test coverage parsing in the project settings under CI/CD/Pipeline settings, but the badge shown on that same page keeps showing unknown.

The documentation is not quite clear to me, as it does not describe the whole procedure. It is clear how to use a badge once created, and there is a manual for publishing a coverage report to pages, but there seems to be no clear path from extracting the score to displaying the badge.

Should I use the coverage entry in my .gitlab-ci.yaml file or fill in the regex in the pipeline settings?

Either way, is Gitlab CI supposed to update the coverage badge based on that, or do I need to use additional tools like coverage-badge to do so?

Where is the extracted coverage score supposed to be reported; how can I find out if my regex works?

Answer 1

I finally got the coverage badge displaying a percentage instead of unknown today for my python project. Here's the relevant content from my .gitlab-ci.yml:

job:
 script:
  - 'python -m venv venv'
  - '.\venv\Scripts\activate'
  - 'python -m pip install -r requirements.txt'
  - 'coverage run --source=python_project -m unittest discover ./tests'
  - 'coverage report --omit=things_that_arent_mine/*'
  - 'coverage xml'
 artifacts:
  reports:
   cobertura: 'coverage.xml'

I'm also using the regex for gcovr listed in the repo CI/CD Settings > General Pipelines > Test coverage parsing which I found after reading this as well as the second to last comment on this:

^TOTAL.*\s+(\d+\%)$

In the repo General Settings > Badges, my badge link is:

http://gitlab-server/%{project_path}/-/jobs

and my badge image url is:

http://gitlab-server/%{project_path}/badges/%{default_branch}/coverage.svg

I don't quite know what the Cobertura report artifact is for (I think it specifically has to do with merge requests) but I have it in there because the tutorials took me down that road. I did confirm that removing the following from the .gitlab-ci.yml doesn't break the badge or coverage number on the jobs page:

  - 'coverage xml'
 artifacts:
  reports:
   cobertura: 'coverage.xml'

While removing or commenting:

- 'coverage report --omit=things_that_arent_mine/*'

does break the badge as well as the coverage number displayed on the CI/CD jobs page of the repo. I also tried some regex variations that I tested in rubular but the only one that didn't cause gitlab to barf was the gcovr one.

Hopefully this helps out, it was kind of difficult and arduous to piece together what was needed for this badge to work on a python project.

EDIT: Also just figured out how to add some sexy precision to the coverage percentage number. If you change the coverage report line of the .gitlab-ci.yml to:

- 'coverage report --omit=things_that_arent_mine/* --precision=2'

And the regex in CI/CD Settings > General Pipelines > Test coverage parsing to:

^TOTAL.+?(\d+.\d+\%)$

That should give you a very precise coverage number that effectively no one but you and I will care about. But by gosh we'll know for sure if coverage is 99.99% or 100%.

Answer 2

A solution that worked for me:

In the .gitlab-ci.yml file, in the job that runs the coverage report, I added the following lines:

  script:
    # some lines omitted for brevity
    - coverage report --omit=venv/*
  coverage: '/TOTAL.*\s+(\d+\%)/'

In particular, I couldn't get the badge to show anything but unknown until I added the coverage directive to my test job. Bear in mind, different tools may print different output and so you will likely have to change the regular expression, as I did.

Answer 3

Spent three days on the problem above myself so I thought I'd post my working config. My project is a pyscaffolding project, uses tox, the pipeline triggers when you push a commit to a branch, and it pushes a pip package to the github packages library.

1. Badge link is: http://gitlab.XXXX.com/XXmeXX/python-template/-/commits/develop

2. Badge image is : http://gitlab.XXXX.com/XXmeXX/python-template/badges/develop/coverage.svg

3. Regex is same as above.

4. PYPIRC is an environment variable set in that looks like a .pypirc file and points to my internal pip registry.

5. I used this tutorial

6. updated -> --cov-report xml is in my setup.cfg and I'm pretty sure b/c of that I don't need the coverage command, but I haven't tested that since I wrote this post. I'll check that next time I'm in there.

My gitlab-ci.yml:

build-package:
  stage: deploy
  image: python:3.7
  script:
   - set
   - cat $PYPIRC > /tmp/.pypirc
   - pip3 install twine setuptools setuptools_scm wheel tox coverage
   # build the pip package
   - python3 setup.py bdist_wheel
   # $CI_COMMIT_TAG only works with the tagging pipeline, if you want to test a branch push directly, pull from fs
   - VERSION=$(python setup.py --version)
   # You can issue ls -al commands if you want, like to see variables or your published packages
   # - echo $VERSION
   # - ls -al ./dist |grep whl
   - tox
   - coverage xml -o coverage.xml
   # If you want to put artifacts up for storage, ...
   # - mkdir public
   - python3 -m twine upload --repository pythontemplate ./dist/my_python_template-${VERSION}-py2.py3-none-any.whl --config-file /tmp/.pypirc
 artifacts:
   reports:
     cobertura : 'coverage.xml'
   when: always
#   If you were copying artifacts up for later.
#  paths:
#     - public
 only:
  - branches

Biggest thing I learned was that first you get it showing as "Coverage" column in the jobs list, then you know you're parsing everything correctly and the regexes are working. From there you work on the coverage xml and the badge links.

Answer 4

I also digged quite a little into this. And just as you said: The command coverage report produces output similar to this:

[...]
tests/__init__.py              0      0   100%
tests/test_ml_squarer.py       4      0   100%
tests/test_squarer.py          4      0   100%
----------------------------------------------
TOTAL                         17      2    88%
test_service run-test: commands[4] | coverage xml

and depending on the regex-expression that is saved, it simply looks for the 88% next to TOTAL. I used the recommend for pytest-cov (Python) , i.e. ^TOTAL.+?(\d+\%)$

So running coverage xml looks rather optional to me at the moment. However, it is not working for me using: GitLab Community Edition 12.10.11

Answer 5

i got this working using an updated yml format.

  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: 'coverage.xml'