Modify CI tests to use bf-pktpy module, with no scapy, except for EBPF backend tests

[jafingerhut]

All prerequisite PRs have been approved and merged, so this PR is ready for review now. All changes to requirements.txt to point at their final "official" commit SHAs are in this PR now.

With these changes, 1 out of the 7 remaining test Python programs that uses Scapy now no longer uses Scapy, leaving only 6 that I believe would take significantly more work to update, and I don't have any plans to change them any time soon. A separate PR proposes licensing those remaining 6 EBPF back end Python test programs as license GPL-2.0-only.

Also with these changes, only the EBPF CI test uses the Scapy library. The rest of the CI tests DO NOT install scapy at all, and they pass. This is good for ensuring that no one accidentally slips in new code that depends on Scapy in the future.

The following prerequisites have already been merged:

• Add bf pktpy source ptf#221
  • This older PR is unnecessary if the one above is merged in: Enable more common bf-pktpy and scapy names to be used through ptf package open-p4studio#85
• Add docs on using pmm (packet manipulation module) bf_pktpy, and option to select pmm by env variable ptf#213
• Replace unnecessary use of Scapy with ptf.pcap_writer in p4tc tests #5165

[jafingerhut]

Since originally writing this comment, I have made the ptf-ebpf tests by installing scapy, but only for the ptf-ebpf tests. I have verified that scapy is not installed for any of the other tests, only bf-pktpy.

@fruffy I am not sure whether there would be any interest in checking in changes similar to this, but basically the all-tests-pass-except the ptf-ebpf test demonstrates that if you do not have scapy installed on your system at all, but you do have the bf-pktpy module installed (which is licensed Apache-2.0), it implements enough functionality that all tests work, both those that do not use ptf at all, and those that do (except the ptf-ebpf tests).

The reason that ptf-ebpf tests fail here is that they import the ptf module, and they import and use scapy directly, too. Some of those ptf-ebpf tests using scapy appear to me unlikely to work with bf-pktpy as an attempted replacement, without probably-significant enhancements to bf-pktpy. I have not attempted to make such enhancements to bf-pktpy, nor do I plan to.

The other passing tests that do use ptf typically only use a handful of functions in ptf, and I believe that only the testgen-p4c-bmv2-ptf tests import the module ptf.packet, which is the only sub-module of ptf that causes either scapy or bf-pktpy to be used. For those tests, bf-pktpy is a compatible replacement, primarily because they do not use much (if any) scapy functionality, and it is a small subset that bf-pktpy does implement. I am not certain, but it appears that the only parts of module ptf that p4testgen PTF tests use are these:

ptf.ptfutils.test_param_get
ptf.ptfutils.send_packet
ptf.ptfutils.verify_packet
ptf.ptfutils.verify_no_other_packets

All packets sent in, or verified as output packets, by testgen-generated PTF tests give the contents of the packets as raw byte strings, so there is no use of any Scapy classes like Ether, IP, IPv6, TCP, UDP, etc. p4testgen is written to be generic to whatever the parser definition is in the P4 program it is generating tests for, so it would be more work to try to use those features of scapy than not to.

[jafingerhut]

One possible reason to consider committing changes like these, is that they would be likely to catch any changes someone tried to make that added dependencies on scapy -- such tests would fail, unless they were part of the ebpf tests.

[fruffy]

Not a big fan of this approach, let's wait until bf_pktpy is default?

Also just noticed, but packet_pktpy appears to be a redundant name.

[jafingerhut]

Regarding making bf_pktpy the default choice for the ptf package: ptf has been around for 10 years, it is used in SONiC testing, and I have no idea how many other places it might be used now. My guess is that changing the default now is going to cause others pain for years to come, and us in the form of issues.

Is there an approach that is not based on environment variables that you are a fan of, that you can think of?

[fruffy]

Can we confirm that this is actually the case? If the behavior of PTF is equivalent with bf_pktpy then downstream repositories should not notice any issues. If not, there is a bug in bf_pktpy. In general, I'd assume these repositories probably pull in scapy separately, like P4C did. Or they pin their PTF version.

[jafingerhut]

Can we confirm whether this will cause other projects pain for years to come, and us in the form of issues? I mean, sure, we can just change the default and find out in the next 2-3 years :-). The thing about a long-public project is that often you do not even know who all of its users are.

[fruffy]

The thing about a long-public project is that often you do not even know who all of its users are.

That is the exact problem to me. We do not have paying customers and users are not providing feedback so we are dealing with an effectively imaginary user base. I rather move on and make changes that are beneficial for the organization and keep complexity low instead of locking us down. We do not have the resources to be backwards compatible.

I also think it is common (good) practice to pin your dependencies, if a project always uses the latest version of software we provide, that's on them. We can change the major version of PTF to signal that it is a breaking change.

[jafingerhut]

OK. Let us imagine for a moment that we do change the default packet manipulation module in the ptf package to bf_pktpy.

In a recent commit today, I tried out running the EBPF tests using ptf with bf_pktpy as the packet manipulation module. It fails, raising exceptions from bf_pktpy about attributes and/or methods that the tests are trying to use not being present.

I am personally not interested in tracking these down and attempting to update bf_pktpy, not even to see how much work it might be. If we want EBPF backend tests to keep working without potentially significant time spent changing them, the most straightforward way is to use scapy as ptf's packet manipulation module for those tests. The way the tests are run right now, that can be done with extra command line options to the ptf command.

I am guessing that there will be other ptf users that will similarly want to continue using Scapy as its packet manipulation module, because of such issues.

Are you OK with recommending to those other people that they use the environment variable PTF_PACKET_MANIPULATION_MODULE="ptf.packet_scapy" to continue using Scapy as their packet manipulation module, if they want to do so?

If not, then I am currently at a loss for how to proceed.

[fruffy]

Are you OK with recommending to those other people that they use the environment variable PTF_PACKET_MANIPULATION_MODULE="ptf.packet_scapy" to continue using Scapy as their packet manipulation module, if they want to do so?

Yes, seems reasonable to me. Like other Python projects we can print a big deprecation warning for a couple months that we are going to switch and then eventually switch with a major PTF release. This gives users enough of a headsup. Does that work?

[jafingerhut]

Sure. And we can look into what it takes to cut a minor release with the current code soon-ish, just before making the backwards-compatible change.

[jafingerhut]

Issue on ptf repo to track this planned change of default: p4lang/ptf#222

[fruffy]

This is the wrong location, the ci build script should have a section for these kinds of tests (PTF_PSA?)