Move non-ECS fields in Network Packet Capture datastream fields out of root namespace

[efd6]

In #5918 it was noted that there is a field definition collision between the Cloud Security Posture integration and the MongoDB datastream in the Network Packet Capture (NPC) integration. This is because both integrations place non-ECS fields in the root namespace.

This appears to be, in the case of NPC, due to Packetbeat predating extensive use of ECS for field definitions.

To address this and the problem more generally, both integrations should move these non-ECS fields into datastream-specific groups. This meta-issue covers the NPC side of this set of changes.

Proposed Approach

Stage 0. (immediately)

Duplicate type into the network.protocol ECS field before all other work.

Notify @brokensound77 when this is done so that detection rules can be updated.

• Completed network_traffic: copy non-ECS fields to ECS-compliant locations for detection rules #8221

Stage 1. (for 8.12)

Non-ECS fields in Network Packet Capture integration (NPC) datastreams will be placed in their own field group (an example of this is seen in a PR here) and generalised packetbeat-related fields will be placed in relevant ECS fields where appropriate (there is no example of this available yet).

Remapping to be performed in a conditional pipeline that user can opt into in the package's configuration UI. Initially the default will leave mappings in the old state, i.e. to not run the remapping pipeline. Clear documentation detailing the impacts of the choice must be included, including how it will impact on detection rules and so on.

Configuration to be per-datastream to allow progressive application of the proposed changes.

Searches and dashboards to be updated to use both the existing field locations and the the new ECS locations to allow users to continue to query/explore data from prior to the changeover.

Equivalent changes to be applied to the Packetbeat ingest pipelines and kibana assets.

• Completed network_traffic: add ECS compatibility pipelines and dashboards #8622, {,x-pack}/packetbeat: add support for pipeline module uploads beats#37291, packetbeat/docs: add documentation for ingest pipelines beats#37798

Stage 2 (for 8.13).

Mark old mappings as deprecated in the package documentation and in the configuration UI at the option to perform the remapping. This documentation to include instructions for users on how to make the change to ECS mappings, and a roadmap/timeline for the deprecation (this issue).

Provide a utility pipeline to convert the mappings of old — pre-change — documents to the new mapping via re-indexing for users who need to maintain historical data after the final deprecation and removal of the old mappings.

• Completed network_traffic: add notice for deprecation of map_to_ecs behaviour #9031

Stage 3 (stage 2. completion plus 6 months).

Change the default of the ECS-mapping pipeline config from off to on, making installations use the ECS mapping while leaving old installations in their existing state.

This stage will require coordination with Fleet (@nimarezainia) to ensure that UI notifications can be made to inform users of the change in behaviour before an upgrade is made.

• Completed [network_traffic] Set map_to_ecs to true by default #10785

Stage 4 (stage 3. completion plus 6 months).

Remove the original mappings and the option to choose which mapping to use. This change to be associated with a major version bump of the integration. Retain old search and dashboard behaviour until stage 5.

This stage will require coordination with Fleet (@nimarezainia) to ensure that UI notifications can be made to inform users of the change in behaviour before an upgrade is made.

BLOCKED: Requires elastic/kibana#187481

• Completed

Stage 5. (optional: no sooner than stage 4. completion plus 6 months)

Remove old search and dashboard behaviour.

This stage may not happen, but if it is decided that it will, will be preceded with clear notification and likely a further major version bump, again via UI notification.

• Completed/Dropped

/cc @siposea @andrewkroh @narph @epixa @nimarezainia @brokensound77

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[nimarezainia]

@kpollich wanted to also bring this to your attention.

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[nimarezainia]

This stage will require coordination with Fleet (@nimarezainia) to ensure that UI notifications can be made to inform users of the change in behaviour before an upgrade is made.

@efd6 these changes are highlighted in the change log correct? could you perhaps open an issue describing what the ask is here. I think the best approach is to have package owners explicitly mention this in the "Compatibility" of the readme for the package, so the information shows up in the integration App and is highlighted for the user to understand and make a decision on.

[efd6]

@nimarezainia The details were to be worked out between fleet and the owners of the integration. We no longer own the integration, but since I wrote the plan, and I see that the step beyond deprecation has caught people unawares, I should raise it as an issue.

[andrewkroh]

@kpollich has opened elastic/kibana#187481 which I think will meet the needs of informing users of the change in behavior when we get to the stage 4 (which will be no earlier than Feb 2025).