How to use forge coverage effectively?

Question

I am trying to get the test coverage for our Smart Contracts. Couple of questions I have.

1. While running the coverage command, it doesn't include all the test cases in the folder, is it because we have written test cases in a modular form, i.e. there are folders inside folders and then test cases, so that the structure remains clean when the codebase grows.

The files included in the report:

The actual files I have:

2. While using the --report summary flag it creates a JSON file that is not practically readable to a human eye, what is the best way to extract the details given in that JSON file?

JSON File :

3. Lastly, are there some docs, etc. for more information on the usage of forge coverage, as the foundry book is pretty limited.

Answer 1

The answers below refer to forge 0.2.0, the latest release at the time of the writing.

foundryup: installed - forge 0.2.0 (2b2a499 2024-01-31T17:41:20.967549332Z)
foundryup: installed - cast 0.2.0 (2b2a499 2024-01-31T17:41:20.972189858Z)
foundryup: installed - anvil 0.2.0 (2b2a499 2024-01-31T17:41:20.965275165Z)
foundryup: installed - chisel 0.2.0 (2b2a499 2024-01-31T17:41:20.969731196Z)

1. Running tests to get the coverage

The coverage operation executes all the tests in the test files, including those in subfolders of the test folder. But you need to put the tests in the test folder (or subfolders) and the source to be tested in the src folder (or subfolders).

When working correctly, what you see in the report generated by the forge coverage command are the results of testing the source code files present in the src folder running the tests in the test folder.

You should not see the file containing the tests listed in that report.

In your case, looking at the screenshot, I see the test cases files (.t.sol) listed in the report, so it means Forge thinks those files must also be included in the coverage report. And as a side note, nothing is executing tests over the test files, so this is the reason why you get 0% for those files.

You need to fix your project structure: remove those test files from being the target of the coverage, put them under a dedicated test folder, move the source code under a src folder, change the importing path accordingly to find the source .sol contracts again.

You project should look something like this:

/src/contracts/PushComm/PushCommV2_5.sol
...
/test/BaseTest.t.sol
/test/PushCore/...
...
/foundry.toml
/README.md
...

Using the src folder as the root folder for your code and the test folder on the same level for the test case is a best practice, but you can change that editing the foundry.toml configuration file; still, I suggest sticking with the best practices to have fewer problems with the repo maintenance.

[profile.default]
src = "src"
out = "out"
libs = ["lib"]

See more config options at the official repo.

2. Coverage reporting options

I'm not sure how you produced that JSON, but the coverage --report option works like this:

• summary: the default output in the form of a table with colors and data
• lcov: coverage in LCOV format; you can use multiple tools to decode that info (i.e., generating HTML reports from that) and also dedicated VS Code plugins like Coverage Gutters
• debug: text format, with some details about the uncovered code parts and what's happening under the hood
• bytecode: it does not work correctly for me, and I see I'm not alone having issues, and this value is not mentioned in the doc even if it's already mentioned in the command options, so I suggest not to use it until it is stable.

3. Documentation and tutorials

Official documentation is not super detailed, but you can find on the Internet articles like this one from Rohanzarathustra, that let you play around with all the possibilities of the coverage command.

Please consider the program is labelled 0.2 for a reason, so expect improvements, changes and breaking changes in the future. ;)

Answer 2

Coverage insights

To complement the answer of Guiseppe Bertone with a concrete command to answer the:

what is the best way to extract the details given in that JSON file?

aspect of your question, the following command generates a report/index.html that allows you to inspect separate functions, and see their line and branch coverage respectively:

clear && forge coverage --report lcov && genhtml -o report --branch-coverage lcov.info

With branch coverage details like:

Coverage and Modular Tests

To answer the:

While running the coverage command, it doesn't include all the test cases in the folder, is it because we have written test cases in a modular form

aspect, I do not know the answer, perhaps forge is updated since then, perhaps you use a different repository configuration. Here is repository a repository that also uses modular tests that are included in the coverage report.

Coverage badge

To address the:

How to use forge coverage effectively?

element of your primary question with a topic that I did not yet find covered a lot for Solidity:

Based on this post the following procedure was generated to include the line- and branch-coverage percentages as a badge in your report using GitHub actions:

0. Create a personal access token in github account, only enable "gist" permission.
1. Go to repo, add that PAT into the repository as new secret named:GIST_SECRET
2. Add the CI.yml to your repo, and write your bash command to get the coverage percentage as environment variable, and store it as GitHub variable. E.g. for forge:

run: echo "BRANCH_COVERAGE=$(awk '/^\| Total/ {print $10}' <<< "$(forge coverage)")" >> $GITHUB_ENV

Then the workflow updates the Gist in here. 3. Reference to this secret gist to include your badge like:

[![Branch coverage badge description][branch-coverage-badge-icon]][coverage_report_link_local]
[branch-coverage-badge-icon]: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/a-t-0/c58317c4d6983cacf14e0466cb1d2438/raw/Decentralised-Saas-Investment-Protocol_branch_coverage.json

4. Full CI.yml that works:

name: "CI"

env:
  API_KEY_ALCHEMY: ${{ secrets.API_KEY_ALCHEMY }}
  FOUNDRY_PROFILE: "ci"

on:
  workflow_dispatch:
  pull_request:
  push:
    branches:
      - "main"

jobs:
  build:
    runs-on: "ubuntu-latest"
    steps:
      - name: "Check out the repo"
        uses: "actions/checkout@v4"

      - name: "Install Foundry"
        uses: "foundry-rs/foundry-toolchain@v1"

      - name: "Install Bun"
        uses: "oven-sh/setup-bun@v1"

      - name: "Install the Node.js dependencies"
        run: "bun install"

      - name: "Build the contracts and print their size"
        run: "forge build --sizes"

      - name: "Add build summary"
        run: |
          echo "## Build result" >> $GITHUB_STEP_SUMMARY
          echo "✅ Passed" >> $GITHUB_STEP_SUMMARY

  test:
    needs: ["build"]
    runs-on: "ubuntu-latest"
    steps:
      - name: "Check out the repo"
        uses: "actions/checkout@v4"

      - name: "Install Foundry"
        uses: "foundry-rs/foundry-toolchain@v1"

      - name: "Install Bun"
        uses: "oven-sh/setup-bun@v1"

      - name: "Install the Node.js dependencies"
        run: "bun install"

      - name: "Show the Foundry config"
        run: "forge config"

      - name: "Generate a fuzz seed that changes weekly to avoid burning through RPC allowance"
        run: >
          echo "FOUNDRY_FUZZ_SEED=$(
            echo $(($EPOCHSECONDS - $EPOCHSECONDS % 604800))
          )" >> $GITHUB_ENV

      - name: "Run the tests"
        run: "forge test"

      - name: "Add test summary"
        run: |
          echo "## Tests result" >> $GITHUB_STEP_SUMMARY
          echo "✅ Passed" >> $GITHUB_STEP_SUMMARY

      - name: "Get the code coverage percentage of the branches as ENV"
        run: echo "LINE_COVERAGE=$(awk '/^\| Total/ {print $4}' <<< "$(forge coverage)")" >> $GITHUB_ENV
      - name: "Get the branch coverage percentage of the branches as ENV"
        # No need to run this twice, you could store the first output.
        run: echo "BRANCH_COVERAGE=$(awk '/^\| Total/ {print $10}' <<< "$(forge coverage)")" >> $GITHUB_ENV

      - name: Get coverage output
        id: coverage
        run: echo "value=$(jq -r '.total.lines.pct|tostring + "%"' coverage/coverage-summary.json)" >> $GITHUB_OUTPUT
      - name: Update code coverage badge JSON gist
        uses: schneegans/[email protected]
        with:
          auth: ${{ secrets.GIST_SECRET }}
          gistID: c58317c4d6983cacf14e0466cb1d2438
          filename: Decentralised-Saas-Investment-Protocol_line_coverage.json
          label: Line Coverage
          message: ${{ env.LINE_COVERAGE }}
          namedLogo: jest
          color: green
          logoColor: lightblue

      - name: Update branch coverage badge JSON gist
        uses: schneegans/[email protected]
        with:
          auth: ${{ secrets.GIST_SECRET }}
          gistID: c58317c4d6983cacf14e0466cb1d2438
          filename: Decentralised-Saas-Investment-Protocol_branch_coverage.json
          label: Branch Coverage
          message: ${{ env.BRANCH_COVERAGE }}
          namedLogo: jest
          color: green
          logoColor: lightblue

And accompanying Readme.md badges:

[![Foundry][foundry-badge]][foundry] [![License: MIT][license-badge]][license]

[![Code coverage badge description][code-coverage-badge-icon]][coverage_report_link_local]

[branch-coverage-badge-icon]: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/a-t-0/c58317c4d6983cacf14e0466cb1d2438/raw/Decentralised-Saas-Investment-Protocol_branch_coverage.json
[coverage_report_link_local]: report/index.html

Yielding, for example:

Disclaimer

I'm involved in writing that repository.