perl core only releases of dual life modules do not show up in CPANSA::DB

[sparrow2009]

Production releases of the perl interpreter at times are shipped with distributions with a version number containing an underscore.

Without any knowledge of the generation process of the CPANSA database it seems, that vulnerable distributions with version numbers interpreted as indicating a development release are not considered for inclusion. This assumption sometimes fails for "core first"-distributions (at least that is how I understand the perl interpreter release process).

Let's take the cored IPC::Cmd module as an example in order to demonstrate the case:

corelist -a IPC::Cmd | grep _
  v5.9.5     0.36_01   
  v5.10.0    0.40_1    
  v5.19.5    0.84_01   
  v5.24.1    0.92_01

cpan-audit dist 'IPC-Cmd' '==0.92_01' > /dev/null; echo $?
0

cpan-audit dist 'IPC-Cmd' '0.92'

advisories_for IPC-Cmd, 0.92
IPC-Cmd (requires 0.92) has 1 advisory
  * CPANSA-IPC-Cmd-2016-1238
    (1) cpan/Archive-Tar/bin/ptar, (2) cpan/Archive-Tar/bin/ptardiff, (3) cpan/Archive-Tar/bin/ptargrep, (4) cpan/CPAN/scripts/cpan, (5) cpan/Digest-SHA/shasum, (6) cpan/Encode/bin/enc2xs, (7) cpan/Encode/bin/encguess, (8) cpan/Encode/bin/piconv, (9) cpan/Encode/bin/ucmlint, (10) cpan/Encode/bin/unidump, (11) cpan/ExtUtils-MakeMaker/bin/instmodsh, (12) cpan/IO-Compress/bin/zipdetails, (13) cpan/JSON-PP/bin/json_pp, (14) cpan/Test-Harness/bin/prove, (15) dist/ExtUtils-ParseXS/lib/ExtUtils/xsubpp, (16) dist/Module-CoreList/corelist, (17) ext/Pod-Html/bin/pod2html, (18) utils/c2ph.PL, (19) utils/h2ph.PL, (20) utils/h2xs.PL, (21) utils/libnetcfg.PL, (22) utils/perlbug.PL, (23) utils/perldoc.PL, (24) utils/perlivp.PL, and (25) utils/splain.PL in Perl 5.x before 5.22.3-RC2 and 5.24 before 5.24.1-RC2 do not properly remove . (period) characters from the end of the includes directory array, which might allow local users to gain privileges via a Trojan horse module under the current working directory.

    Affected range: <0.96
    Fixed range:    >=0.96

    CVEs: CVE-2016-1238

    References:
    http://www.nntp.perl.org/group/perl.perl5.porters/2016/07/msg238271.html
    http://www.securitytracker.com/id/1036440
    http://perl5.git.perl.org/perl.git/commit/cee96d52c39b1e7b36e1c62d38bcd8d86e9a41ab
    https://rt.perl.org/Public/Bug/Display.html?id=127834
    http://www.securityfocus.com/bid/92136
    http://www.debian.org/security/2016/dsa-3628
    https://h20566.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay?docId=emr_na-c05240731
    https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/DOFRQWJRP2NQJEYEWOMECVW3HAMD5SYN/
    https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/TZBNQH3DMI7HDELJAZ4TFJJANHXOEDWH/
    https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/2FBQOCV3GBAN2EYZUM3CFDJ4ECA3GZOK/
    https://security.gentoo.org/glsa/201701-75
    https://lists.apache.org/thread.html/7f6a16bc0fd0fd5e67c7fd95bd655069a2ac7d1f88e42d3c853e601c@%3Cannounce.apache.org%3E
    https://lists.debian.org/debian-lts-announce/2018/11/msg00016.html
    https://security.gentoo.org/glsa/201812-07
    http://lists.opensuse.org/opensuse-security-announce/2019-08/msg00002.html

This is a byproduct of the belated reviewing of your version of pull request briandfoy/cpan-audit#62 requested by you. Expect some patches and comments there in the following weeks.

[briandfoy]

Okay, I see this issue, and I suspect that it's going to take some major surgery to fix this.

For IPC::Cmd, the version 0.92_01 is not indexed on CPAN so it does not show up in the list of versions that MetaCPAN::Client will return, and it thus does not show up in the list of versions in the CPANSA::DB data structure.

I'll have to think about this. Module::Corelist can probably help. This would be in the util/generate script.

Despite what should happen, we still have to respond to it. Some other things to complain about:

• perl should not distribute development versions
• any dual-lived module should release all version to CPAN
• MetaCPAN should include versions only distributed with Perl.

[briandfoy]

I've been looking at this and the data structure that everything expects has no way to handle this. But, this is a symptom of a larger problem. A module can show up in different distributions through its life, so mapping a package name to a dist isn't that simple. You have to map a (package,version) pair to a dist. And, for dual-lived modules, that (package,version) pair can map to multiple distributions.

I think the first thing that needs to change is module2dist part of the data structure and the parts of the code that use that. Maybe there's something that given a package name, returns a list of versions as keys and an array ref of dists as values.

For dual-lived modules, something needs to recognize the namespace is in one distribution by its reports are cataloged in another.

But maybe we can cheat by looking through corelist that injecting versions we find there into the version list in their CPAN packages. Go through the list of versions for a dist, see if every version in corelist is there, and add ones that aren't. That doesn't solve the bigger problem though.

[briandfoy]

I'm able to add some corelist versions to the version list for a dist in the CPANSA::DB. However, this is imperfect. For example, App::Cpan is in core, but I have to work backward to get to CPAN as the dist name and take the corelist version of CPAN.pm. I have to guess that the version of the main module is also the dist version. Everything is about dist versions, not individual module versions. A harder example is a distribution that is not a module name, such as PathTools. Its main module is Cwd.

The next thing is making these new entries useable. I think it works for this issue, using the unreleased versions I have locally (from 846691d). I'm not confident this is solid yet so I'm going to play with it a bit more.

$ perl -Ilib script/cpan-audit dist IPC-Cmd ==0.92_01
IPC-Cmd (requires ==0.92_01) has 1 advisory
  * CPANSA-IPC-Cmd-2016-1238
    (1) cpan/Archive-Tar/bin/ptar, (2) cpan/Archive-Tar/bin/ptardiff, (3) cpan/Archive-Tar/bin/ptargrep, (4) cpan/CPAN/scripts/cpan, (5) cpan/Digest-SHA/shasum, (6) cpan/Encode/bin/enc2xs, (7) cpan/Encode/bin/encguess, (8) cpan/Encode/bin/piconv, (9) cpan/Encode/bin/ucmlint, (10) cpan/Encode/bin/unidump, (11) cpan/ExtUtils-MakeMaker/bin/instmodsh, (12) cpan/IO-Compress/bin/zipdetails, (13) cpan/JSON-PP/bin/json_pp, (14) cpan/Test-Harness/bin/prove, (15) dist/ExtUtils-ParseXS/lib/ExtUtils/xsubpp, (16) dist/Module-CoreList/corelist, (17) ext/Pod-Html/bin/pod2html, (18) utils/c2ph.PL, (19) utils/h2ph.PL, (20) utils/h2xs.PL, (21) utils/libnetcfg.PL, (22) utils/perlbug.PL, (23) utils/perldoc.PL, (24) utils/perlivp.PL, and (25) utils/splain.PL in Perl 5.x before 5.22.3-RC2 and 5.24 before 5.24.1-RC2 do not properly remove . (period) characters from the end of the includes directory array, which might allow local users to gain privileges via a Trojan horse module under the current working directory.