Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent indexing of event.received field #11602

Closed
wants to merge 12 commits into from
Closed

Prevent indexing of event.received field #11602

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c5aba59
remove event.received from app_logs
endorama Sep 7, 2023
050b238
remove event.received from app_metrics
endorama Sep 7, 2023
f125168
remove event.received from error_logs
endorama Sep 7, 2023
663e333
remove event.received from internal_metrics
endorama Sep 7, 2023
f58d6c4
remove event.received from rum_traces
endorama Sep 7, 2023
aa5e4ea
remove event.received from traces
endorama Sep 7, 2023
cbd8394
update changelog
endorama Sep 7, 2023
aac6581
add PR link to changelog
endorama Sep 7, 2023
8ac04bd
remove removeEventReceivedBatchProcessor
endorama Sep 7, 2023
20051e9
add remove_event_received to genpackage
endorama Sep 7, 2023
8e51d6b
use shared pipeline
endorama Sep 7, 2023
cee297b
changelog: is enhancement
endorama Sep 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions apmpackage/apm/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
- description: Remove unused `processor.event` field from logs data streams
type: enhancement
link: https://github.com/elastic/apm-server/pull/11494
- description: Remove `event.received` fields from logs, metrics and traces data streams
type: bugfix
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type: bugfix
type: enhancement

There's no bug in apmpackage (mapping and ingest pipeline) itself

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, followed up in cee297b

link: https://github.com/elastic/apm-server/pull/11602
- version: "8.10.0"
changes:
- description: Add permissions to reroute to dedicated datasets for logs, metrics and traces
Expand Down
5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/app_logs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ processors:
name: observer_ids
- pipeline:
name: ecs_version
- remove:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better, define a pipeline like the one in https://github.com/elastic/apm-server/blob/main/apmpackage/cmd/genpackage/pipelines.go#L155 so that it can be reused in multiple ingest pipelines.

field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.
5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,8 @@ processors:
name: client_geoip
- pipeline:
name: set_metrics
- remove:
field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.
5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/error_logs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,8 @@ processors:
if: ctx.error?.log?.message != null
field: error.grouping_name
copy_from: error.log.message
- remove:
field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.
5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/internal_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,8 @@ processors:
- remove:
field: _dynamic_templates
ignore_missing: true
- remove:
field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.
5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/rum_traces/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,9 @@ processors:
- service.framework
ignore_missing: true
ignore_failure: true
- remove:
field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.

5 changes: 5 additions & 0 deletions apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,8 @@ processors:
value: 0
ignore_failure: true
# end of event.success_count logic
- remove:
field: event.received
ignore_missing: true
ignore_failure: true
description: The field is used for internal statistics but does not need to be indexed.