Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

10631: Get CircleCI to store Python coverage report #10640

Merged

Conversation

stevejalim
Copy link
Collaborator

@stevejalim stevejalim commented Oct 25, 2021

Description

Now we have coverage.py running again, we want to surface the metrics at CI level.

This changeset stores the output of the coverage checks as a CircleCI artefact.

We then use CodeCov to report on the result. The CodeCov integration is powered by their own Orb, which runs verification checks on the uploader script to ensure its integrity - you can see those steps happen in CircleCI

Issue / Bugzilla link

#10631

Testing

  • This branch should go through CircleCI for tests and result in a python_coverage dir existing in the artefacts for the build.
  • This resulting coverage data should be piped via CodeCov and trigger a custom message, below, reporting current coverage status
  • There should be no failure conditions tied to coverage for now - let's leave humans decide
@stevejalim stevejalim force-pushed the 10631--python-coverage-reports-in-prs branch 5 times, most recently from 59d0624 to 88d7ab1 Compare October 26, 2021 22:09
@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (master@35e60ae). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master   #10640   +/-   ##
=========================================
  Coverage          ?   69.42%           
=========================================
  Files             ?      149           
  Lines             ?     8402           
  Branches          ?        0           
=========================================
  Hits              ?     5833           
  Misses            ?     2569           
  Partials          ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35e60ae...88d7ab1. Read the comment docs.

@stevejalim
Copy link
Collaborator Author

@robhudson @pmac So we've got coverage metrics being pumped back into the PR. This one lacks a base commit to compare to, so we might find we need to do a bit of tuning, but there is (deliberately) no config that fails CI or similar based on a change in coverage so I (famous last words) think we won't get any blocked PRs because of this. We can look to add strict checks in the future, if we decide it's wortwhile.

@stevejalim stevejalim force-pushed the 10631--python-coverage-reports-in-prs branch from 88d7ab1 to 2c6e39c Compare October 26, 2021 22:32
@stevejalim
Copy link
Collaborator Author

Am 95% sure that the codecov.yml provided will show us a coverage chart every time CI completes, but I see that the one in the PR above hasn't updated, so I will keep an eye on this aspect once this is merged in case it needs tuning

@stevejalim stevejalim force-pushed the 10631--python-coverage-reports-in-prs branch from 2c6e39c to 296dc1e Compare October 26, 2021 22:51
Copy link
Member

@pmac pmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Let's give it a try 🍰

@stevejalim stevejalim merged commit 65d7663 into mozilla:master Oct 27, 2021
@robhudson
Copy link
Member

Excited to start seeing these!

@stevejalim stevejalim deleted the 10631--python-coverage-reports-in-prs branch November 1, 2021 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants