Python coverage support broken (v0.6.10 of coveralls report problem)

[set-soft]

Yesterday the "coveralls report" 0.6.10 was released. This started to generate the following error:

🔍 Detected coverage file: .coverage
⚠️ Coverage reporter does not yet know how to process this file: .coverage
🚨 Nothing to report

Forcing the format to python I got:

✏️ Forced coverage format: python
🔍 Detected coverage file: .coverage
#<CoverageReporter::CoveragepyParser::ParserError:There was an error processing .coverage: --: 1: coverage: not found


To use the python format, do one of the following:
1. Make sure that the coverage executable is available in the
   runner environment, or
2. Convert the .coverage file to a coverage.xml file by running
   `coverage xml`. Then pass the input option `format: cobertura`
   (for Coveralls GitHub Action or orb), or pass `--format=cobertura`
   if using the coverage reporter alone.

The first option seems to be impossible for an action. So I tried the second. Now I get:

✏️ Forced coverage format: cobertura
🔍 Detected coverage file: coverage.xml
🔍 Detected coverage file: .coverage
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 8752691638`
#<IO::Error:Error reading file: Is a directory>

I still trying things, but I think the action should fix it.

[MarcelGeo]

Hello there. I have same issue :) This is caused by PR coverallsapp/coverage-reporter#122 , but not sure - how to fix that :)

[set-soft]

I dropped the action and I'm just using coveralls reporter 0.6.9, which works for Python. I couldn't get 0.6.10 working, I tried using the recommended XML export and also installing the Python coverage tool, always failed.

I use something like this:

    - name: Upload Coverage optional
      env:
        COVERALLS_DEBUG: true
        COVERALLS_FLAG_NAME: ${{ matrix.ki_release }}_${{ matrix.w_tests }}
        COVERALLS_PARALLEL: true
        COVERALLS_ENDPOINT: https://coveralls.io
        COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        COVERALLS_SOURCE_HEADER: github-action
      run: |
        mkdir -p ~/bin/
        echo ~/bin >> $GITHUB_PATH
        export CURDIR=`pwd`
        cd ~/bin/
        curl -sLO https://github.com/coverallsapp/coverage-reporter/releases/download/v0.6.9/coveralls-linux.tar.gz
        curl -sLO https://github.com/coverallsapp/coverage-reporter/releases/download/v0.6.9/coveralls-checksums.txt
        cat coveralls-checksums.txt | grep coveralls-linux.tar.gz | sha256sum --check
        tar -xzf coveralls-linux.tar.gz
        rm coveralls-checksums.txt
        cd ${CURDIR}
        ~/bin/coveralls report --debug --format python .coverage

[afinetooth]

Hi @set-soft. It looks like the issue might be with your last line.

That is to say, there is an issue with your last line that will cause an error: --format python. Right now, the direct parsing of pytest-cov (ie. .coverage files is not supported). So if you want to use the XML conversion step, coverage xml, you'll want to use --format cobertura, since the XML format is Cobertura XML.

(--format python will be supported soon, but it will require having coverage.py in your environment, so we can perform the conversion step for you.)

Let me know if you have any luck with 0.6.10, after converting your .coverage file to Cobertura XMl with coverage xml and using --format cobertura.

It's great that you're using the --debug flag. If the above doesn't work, please share the debug output from your CI build log and we'll go from there.

[set-soft]

Hi @set-soft. It looks like the issue might be with your last line.

That is to say, there is an issue with your last line that will cause an error: --format python. Right now, the direct parsing of pytest-cov (ie. .coverage files is not supported). So if you want to use the XML conversion step, coverage xml, you'll want to use --format cobertura, since the XML format is Cobertura XML.

I also tried this (before python), didn't work.

(--format python will be supported soon, but it will require having coverage.py in your environment, so we can perform the conversion step for you.)

Let me know if you have any luck with 0.6.10, after converting your .coverage file to Cobertura XMl with coverage xml and using --format cobertura.

When I tried using cobertura I got

  v0.6.10

✏️ Forced coverage format: cobertura
🔍 Detected coverage file: coverage.xml
🔍 Detected coverage file: .coverage
⭐️ Running in parallel mode. You must call the webhook after all jobs finish: `coveralls done --build-number 8752691638`
#<IO::Error:Error reading file: Is a directory>

You can see the run here

I also tried removing the .coverage file here

It's great that you're using the --debug flag. If the above doesn't work, please share the debug output from your CI build log and we'll go from there.

Lamentably I didn't have debug enabled when I tried cobertura.

[afinetooth]

Hi @set-soft. I submitted this as a potential issue with 0.6.10 not ignoring files while searching for file extensions (including .coverage in the context of a directory in your environment, .coverage/).

If you're concerned about getting it to work before we can have a look, I suggest renaming that directory if feasible.

I'll update you here when I hear back.

[afinetooth]

@set-soft. Follow-up clarification from my team member:

When you said:

I also tried removing the .coverage file here

Can I confirm that you mean you removed any and all .coverage files from your environment?

Specifically, did you remove the .coverage file but leave a .coverage/ directory in your environment?

Thanks 🙏

[set-soft]

Hi @afinetooth !

Take a look at the Upload Coverage optional (requisite) step, it does rm .coverage.

So .coverage was a file, not a directory, and was removed to avoid confusion.

You can see the output of the ls command at the end of the step Collect coverage ki8 it says:

-rw-r--r-- 1 root root   77824 Apr 19 11:47 .coverage
-rw-r--r-- 1 root root     371 Apr 19 11:42 .coveragerc
-rw-r--r-- 1 root root 1071161 Apr 19 11:47 coverage.xml

This step ran:

        python3-coverage combine
        python3-coverage report
        python3-coverage html -d output/htmlcov_${{ matrix.w_tests }}
        python3-coverage xml
        ls -la coverage* .coverage*

I never knew what directory was the problem.

I tried running the same docker image in my local machine and tried to reproduce the steps, I got:

 
⠀⠀⠀⠀⠀⠀⣿
⠀⠀⠀⠀⠀⣼⣿⣧⠀⠀⠀⠀⠀⠀⠀ ⣠⣶⣾⣿⡇⢀⣴⣾⣿⣷⣆ ⣿⣿⠀⣰⣿⡟⢸⣿⣿⣿⡇ ⣿⣿⣿⣷⣦⠀⠀⢠⣿⣿⣿⠀⠀⣿⣿⠁⠀⣼⣿⡇⠀⢀⣴⣾⣿⡷
⠶⣶⣶⣶⣾⣿⣿⣿⣷⣶⣶⣶⠶  ⣸⣿⡟ ⠀⢠⣿⣿⠃⠈⣿⣿⠀⣿⣿⢠⣿⡿⠀⣿⣿⣧⣤⠀⢸⣿⡇⣠⣿⡿⠀⢠⣿⡟⣿⣿⠀⢸⣿⡿⠀⠀⣿⣿⠃⠀⢸⣿⣧⣄
⠀⠀⠙⢻⣿⣿⣿⣿⣿⡟⠋⠁⠀⠀ ⣿⣿⡇⠀ ⢸⣿⣿⠀⣸⣿⡟⠀⣿⣿⣾⡿⠁ ⣿⣿⠛⠛⠀⣿⣿⢿⣿⣏⠀⢀⣿⣿⣁⣿⣿⠀⣾⣿⡇⠀⢸⣿⡿⠀⠀⡀⠙⣿⣿⡆
⠀⠀⢠⣿⣿⣿⠿⣿⣿⣿⡄⠀⠀⠀ ⠙⢿⣿⣿⠇⠈⠿⣿⣿⡿⠋⠀⠀⢿⣿⡿⠁⠀⢸⣿⣿⣿⡇⢸⣿⣿⠀⣿⣿⣄⣾⣿⠛⠛⣿⣿⢠⣿⣿⣿ ⣼⣿⣿⣿ ⣿⣿⡿⠋⠀
⠀⢀⣾⠟⠋⠀⠀⠀⠙⠻⣷⡀⠀⠀
 
  v0.6.10

✏️ Forced coverage format: cobertura
🔍 Detected coverage file: coverage.xml
⏱️ Report parsing: 00:00:00.355358404
⏱️ Report request: 00:00:00.005574598
Unhandled exception: Error reading file: Is a directory (IO::Error)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???

I used coveralls report --format cobertura -r pp --debug

[altaurog]

I just encountered this. I was able to work around it by specifying --cov-report=lcov to pytest.

using lcov format also produced correct results, whereas with the default .coverage output I had observed a discrepancy in the reported coverage which seemed to stem from improper parsing of the python files.

[afinetooth]

Sorry guys, I've been OOO.

@altaurog, thanks for sharing your solution.

@set-soft some good news, this PR should fix your issue and will be released soon as v. 0.6.11 of Coverage Reporter, the official integration underlying the Coveralls GitHub Action.

Until then, some potential mitigating actions you can take:

1. Pin the version of Coverage Reporter that the GitHub Action uses. You can do that with the latest version (v2.3.0) of the action. I'd suggest starting with v0.6.9 of the reporter.
2. You should be able to remove your .coverage file from the local environment before the upload step. I don't know why that's not working but can I ask a question?:

You said you ran this:

coveralls report --format cobertura -r pp --debug

Csn you tell me why you used the -r pp flag? I'm not familiar with that.

[afinetooth]

Cleaning house. Gonna close this issue as it should have been resolved by coverage-reporter PR: coverallsapp/coverage-reporter#125.

If anyone disagrees, or is still having issues with python (or cobertura format reports exported by coverage.py via coverage xml), just re-open or just ping me here.

[MarcelGeo]

I added lcov format as output of pytest and now it's running. I'm not sure, why it was running before 0.6.10. Without lcov, no changes are detected.

MerginMaps/server#311

[afinetooth]

@MarcelGeo yes, lcov works perfectly fine as well.

In general, we recommend exporting in Cobertura XML format, where the lines would be slightly different from your changes, as follows:

          [...]
          pipenv run pytest -v --cov=mergin --cov-report=xml mergin/tests

      - name: Coveralls
        uses: coverallsapp/github-action@v2
        with:
          base-path: server
          format: cobertura

In v0.6.9 and v0.6.10 of coverage-reporter, we were trying to resolve some issues our pytest-cov parser (in beta) was having while directly parsing .coverage files, so we resolved those by looking for the coverage binary in your environment, if a .coverage file was present, using it to convert the .coverage report to cobertura XML, with coverage xml; or, if there were other formats available, whether cobertura or lcov, going ahead and parsing them.

Prior to that, and for a short time during v0.6.9 it's possible that coverage-reporter (the integration underlying our github-action) failed to parse, or even failed to recognize, your local .coverage file. (You may see that it threw an error to that effect if you look back at your build log for CI build.)

In any case, we're on v0.6.15 of coverage-reporter now, and v2.3.4 of the github-action, and you can always use the latest version of the github-action by calling it with @v2, like:

    - name: Coveralls
      uses: coverallsapp/github-action@v2
      [...]