Skip to content

[cloudflare_logpush] Ingest pipeline improvements#18257

Open
brijesh-elastic wants to merge 18 commits intoelastic:mainfrom
brijesh-elastic:cloudflare_logpush-pipeline
Open

[cloudflare_logpush] Ingest pipeline improvements#18257
brijesh-elastic wants to merge 18 commits intoelastic:mainfrom
brijesh-elastic:cloudflare_logpush-pipeline

Conversation

@brijesh-elastic
Copy link
Copy Markdown
Collaborator

@brijesh-elastic brijesh-elastic commented Apr 7, 2026 •
edited
Loading

Proposed commit message

cloudflare_logpush: ingest pipeline improvements

Enhancements:
- Updated the integration to use ECS version 9.3.0 and format_version 3.3.2.
- Added missing convert processors across all data streams. This ensures fields declared as ip, long,
boolean, or double in fields.yml are correctly typed.
- Added support for new fields across 9 data streams (device_posture, gateway_dns, gateway_http,
gateway_network, http_request, network_analytics, network_session, workers_trace, firewall_event).
- Sorted all fields.yml entries alphabetically for better maintainability.
- Converted grok processors to dissect for improved performance.
- Consolidated multiple timestamp-to-Unix-millis script processors into a single, efficient script.
- Implemented the latest null removal script across all data streams.
- Updated error.message values to use the full, standardized format.
- Added a tag key to every processor for easier debugging.
- Removed ignore_failure: true from the initial JSON processor.
- Implemented dynamic mapping for dns.response_code and dns.question.type to follow IANA keyword
representations in dns and dns_firewall data streams.

Bugfixes:
- email_security_alerts - Fixed timestamp normalization by correcting the Painless script to reference
ctx.json.Timestamp (PascalCase) instead of ctx.json.timestamp.
- firewall_event - Resolved swapped field descriptions for origin.ray.id and origin.response.status
- http_request - Resolved swapped field descriptions for cache.status and cache.response.status
- http_request - Aligned header naming between the pipeline and fields.yml (changed singular header
to plural headers) (for RequestHeaders/ResponseHeaders).
- spectrum_event - Fixed a broken grok guard condition that used an incorrect path and a
tautological || operator. Also corrected the remove processor to reference action instead of event_action.
- network_analytics - Fixed case-sensitivity in the split processor condition (TCPSackBlocks).
- audit - Corrected the rename condition to use ctx.json.Interface (PascalCase).
- Fixed instances where fields documented as integers or arrays of integers were incorrectly
mapped. Replaced simple rename processors with convert processors to ensure these
fields are correctly cast to string to match the `type: keyword` definition in fields.yml.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/cloudflare_logpush directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues

@brijesh-elastic brijesh-elastic self-assigned this Apr 7, 2026
@brijesh-elastic brijesh-elastic requested a review from a team as a code owner April 7, 2026 15:19
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:cloudflare_logpush Cloudflare Logpush Category: Integration quality Category: Quality used for SI planning Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Apr 7, 2026
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 7, 2026

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026 •
edited
Loading

Vale Linting Results

Summary: 100 warnings, 9 suggestions found

⚠️ Warnings (100)
File Line Rule Message
packages/cloudflare_logpush/docs/README.md 108 Elastic.DontUse Don't use 'please'.
packages/cloudflare_logpush/docs/README.md 112 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 119 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 131 Elastic.DontUse Don't use 'just'.
packages/cloudflare_logpush/docs/README.md 141 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 152 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 153 Elastic.DontUse Don't use 'Please'.
packages/cloudflare_logpush/docs/README.md 154 Elastic.DontUse Don't use 'please'.
packages/cloudflare_logpush/docs/README.md 194 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 350 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 352 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 353 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 354 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 503 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 505 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 506 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 507 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 676 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 678 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 679 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 680 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 856 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 858 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 859 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 860 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 961 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'that is' instead of 'i.e'.
packages/cloudflare_logpush/docs/README.md 963 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 965 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 966 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 967 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1091 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 1093 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 1094 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1095 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1246 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 1257 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 1259 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 1260 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1261 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1517 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 1519 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 1520 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1521 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1749 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 1751 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 1752 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 1753 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2092 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 2094 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 2095 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2096 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2348 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'eg'.
packages/cloudflare_logpush/docs/README.md 2377 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 2393 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 2395 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 2396 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2397 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2606 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 2617 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 2619 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 2620 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 2621 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3064 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 3066 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 3067 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3068 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3231 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 3233 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 3234 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3235 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3345 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 3347 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 3348 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3349 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3701 Elastic.BritishSpellings Use American English spelling 'acknowledgment' instead of British English 'Acknowledgement'.
packages/cloudflare_logpush/docs/README.md 3720 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 3722 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 3723 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3724 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 3985 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/cloudflare_logpush/docs/README.md 4004 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 4006 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 4007 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4008 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4127 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 4129 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 4130 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4131 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4339 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 4341 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 4342 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4343 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4544 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 4546 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 4547 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4548 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4697 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/cloudflare_logpush/docs/README.md 4699 Elastic.QuotesPunctuation Place punctuation inside closing quotation marks.
packages/cloudflare_logpush/docs/README.md 4700 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/cloudflare_logpush/docs/README.md 4701 Elastic.Latinisms Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
💡 Suggestions (9)
File Line Rule Message
packages/cloudflare_logpush/docs/README.md 5 Elastic.Semicolons Use semicolons judiciously.
packages/cloudflare_logpush/docs/README.md 5 Elastic.Semicolons Use semicolons judiciously.
packages/cloudflare_logpush/docs/README.md 5 Elastic.Semicolons Use semicolons judiciously.
packages/cloudflare_logpush/docs/README.md 136 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/cloudflare_logpush/docs/README.md 184 Elastic.WordChoice Consider using 'refer to if it's a document, view if it's a UI element' instead of 'See', unless the term is in the UI.
packages/cloudflare_logpush/docs/README.md 667 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/cloudflare_logpush/docs/README.md 1713 Elastic.WordChoice Consider using 'top-level' instead of 'first-class', unless the term is in the UI.
packages/cloudflare_logpush/docs/README.md 2359 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/cloudflare_logpush/docs/README.md 3060 Elastic.Wordiness Consider using 'whether' instead of 'Whether or not'.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 8, 2026 •
edited
Loading

🚀 Benchmarks report

Package cloudflare_logpush 👍(10) 💚(3) 💔(8)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
gateway_http 3649.64 2898.55 -751.09 (-20.58%) 💔
gateway_network 6756.76 5681.82 -1074.94 (-15.91%) 💔
nel_report 29411.76 19607.84 -9803.92 (-33.33%) 💔
network_session 4366.81 2057.61 -2309.2 (-52.88%) 💔
sinkhole_http 6896.55 5050.51 -1846.04 (-26.77%) 💔
casb 4807.69 3378.38 -1429.31 (-29.73%) 💔
dns 18518.52 7407.41 -11111.11 (-60%) 💔
email_security_alerts 2493.77 1811.59 -682.18 (-27.36%) 💔

To see the full report comment with /test benchmark fullreport

efd6
efd6 reviewed Apr 9, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is too much going on here. It's good that the changes are broken out in to distinct commits, but those commits do not explain what they are doing beyond the subject line. If we need to come back to this change to understand why a particular change was made in order to handle a support case, it will be essentially impossible to understand why any given decision was made.

@brijesh-elastic
update ECS version to 9.3.0 and format_version to 3.3.2
12b5f8f 
Update format_version to 3.3.2 and ECS dependency to git@v9.3.0 in
manifest.yml and build.yml. Update ecs.version to 9.3.0 in all 21
data stream ingest pipelines.
@brijesh-elastic
add new fields, sort fields.yml, update ECS field definitions, and fix
d10d0d0 
field descriptions

Update ECS field definitions by replacing agent.yml with beats.yml and
modernizing base-fields.yml across all 21 data streams.

Sort all fields.yml entries alphabetically for better maintainability.
Fix swapped field descriptions for firewall_event
(origin.ray.id/origin.response.status) and http_request
(cache.status/cache.response.status).

Add support for new fields across 9 data streams with their
corresponding ingest pipeline processors: device_posture
(RegistrationID), firewall_event (FraudUserID), gateway_dns (12 fields
including InternalDNS*, QueryApplication*, RequestContext*),
gateway_http (AppControlInfo, ApplicationStatuses, RedirectTargetURI,
RegistrationID), gateway_network (RegistrationID), http_request (11
fields including Fraud*, WebAssets*, WorkerScriptName), network_analytics
(DNSQueryName, DNSQueryType, PFPCustomTag), network_session
(InitialOriginIP, RegistrationID, ResolvedFQDN, SNI), workers_trace
(CPUTimeMs, WallTimeMs).
@brijesh-elastic
fix email_security_alerts timestamp normalization
0fe3c67 
Correct the Painless script to reference ctx.json.Timestamp (PascalCase)
instead of ctx.json.timestamp, matching the actual field name from the
Cloudflare API and the guard condition.
@brijesh-elastic
fix spectrum_event broken grok guard condition and remove processor
42f0f65 
Fix the grok guard condition that used an incorrect path
(ctx.json?.cloudflare_logpush) instead of (ctx.cloudflare_logpush) and
a tautological || operator instead of &&. Also correct the remove
processor to reference action instead of event_action. Update test
data to use a valid disconnect timestamp.
@brijesh-elastic
fix network_analytics split processor case-sensitivity for TCPSackBlocks
e75e2a3 
Correct the split processor condition to reference ctx.json.TCPSackBlocks
consistently instead of mixing TCPSACKBlocks and TCPSackBlocks casing.
@brijesh-elastic
fix audit rename condition to use PascalCase for Interface field
6c31760 
Correct the rename condition to use ctx.json?.Interface (PascalCase)
matching the actual Cloudflare API field name instead of lowercase.
@brijesh-elastic
fix integer-to-keyword type casting by replacing rename with convert
03d0a3f 
Replace rename processors with convert processors (type: string) for
fields documented as integers or arrays of integers but mapped as keyword
type in fields.yml. Affected fields: gateway_dns (CNAMECategoryIDs,
EDEErrors, InitialCategoryIDs, MatchedIndicatorFeedIDs,
ResolvedIPCategoryIDs), gateway_http (ApplicationIDs), gateway_network
(ApplicationIDs, CategoryIDs).
@brijesh-elastic
align http_request header naming between pipeline and fields.yml
7835b9e 
Change singular header to plural headers for RequestHeaders and
ResponseHeaders target fields to match the fields.yml definitions
(request.headers and response.headers).
@brijesh-elastic
implement dynamic DNS mapping for dns and dns_firewall data streams
d4aa022 
Add IANA keyword representation scripts for dns.response_code and
dns.question.type in both dns and dns_firewall data streams. Numeric
DNS response codes are now mapped to human-readable names (e.g., 0 ->
NoError, 3 -> NXDomain) and query types are mapped to their IANA
names (e.g., 1 -> A, 28 -> AAAA, 15 -> MX).
@brijesh-elastic
remove ignore_failure from initial JSON processor
328e06c 
Remove ignore_failure: true from the first JSON processor in all data
stream pipelines that had it. Parsing failures should surface as errors
rather than silently producing partial documents.
@brijesh-elastic
convert grok processors to dissect for improved performance
a66ab5d 
Replace grok processors with dissect for simple delimiter-based pattern
matching in firewall_event, http_request, and spectrum_event data
streams. Dissect is more performant than grok for fixed patterns like
protocol/version splitting.
@brijesh-elastic
consolidate timestamp handling across all data streams
71f2c5e 
Replace multiple timestamp normalization scripts (which handled both
String and Number types with try/catch) with a single, efficient script
that only handles Number type. The new script directly converts
timestamps to Unix milliseconds by dividing nanosecond values or
multiplying second values.

Update test input log files to use numeric timestamps to match the
simplified script expectations.
@brijesh-elastic
add missing convert processors across all data streams
cd94280 
Replace rename processors with typed convert processors for fields
declared as ip, long, boolean, or double in fields.yml to ensure
correct type casting. Add in-place convert processors for timestamp
fields to handle string-to-number conversion before the normalization
script.

Affected data streams: access_request, device_posture, dns, dns_firewall,
gateway_dns, gateway_http, gateway_network, http_request, magic_ids,
network_session, sinkhole_http, spectrum_event. Also adds timestamp
converts for all 20 data streams with numeric timestamp handling.
@brijesh-elastic
implement null removal script and standardize error.message format
a2e8333 
Add a null/empty field removal script at the end of all 21 data stream
pipelines. The script recursively removes fields with null values, empty
strings, empty maps, and empty lists.

Standardize all on_failure error.message values to use the full format:
"Processor {type} with tag {tag} in pipeline {pipeline} failed with
message: {message}" for consistent debugging output.
@brijesh-elastic
add tags for every processor across all data streams
a33558a 
Add a unique tag key to every processor in all 21 ingest pipelines for
easier debugging and tracing of pipeline failures. Tags follow the
pattern: {processor_type}_{field_description}_{hash}.
@brijesh-elastic
add changelog entry for cloudflare_logpush pipeline improvements
b735eab
@brijesh-elastic
run pipeline and system tests
15a1e7f
@brijesh-elastic brijesh-elastic force-pushed the cloudflare_logpush-pipeline branch from 2dc01c1 to 15a1e7f Compare April 12, 2026 10:32
macroscopeapp[bot]
macroscopeapp bot reviewed Apr 12, 2026
View reviewed changes
...ages/cloudflare_logpush/data_stream/firewall_event/elasticsearch/ingest_pipeline/default.yml
copy_from: cloudflare_logpush.firewall_event.client.request.method
ignore_empty_value: true
- convert:
tag: convert_json_EdgeResponseStatus_to_cloudflare_logpush_firewall_event_edge_response_status_8ff95d40
Copy link
Copy Markdown

@macroscopeapp macroscopeapp bot Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium ingest_pipeline/default.yml:135

The convert processors for json.EdgeResponseStatus, json.ClientASN, json.MatchIndex, and json.OriginResponseStatus attempt to convert empty strings to long, which fails and triggers pipeline_error in event.kind even though these are valid optional fields. Previously these had if: ctx.json?.field != '' guards that were removed. Consider restoring the empty string checks to prevent valid documents with empty optional fields from being marked as errors.

Also found in 1 other location(s)

packages/cloudflare_logpush/data_stream/nel_report/elasticsearch/ingest_pipeline/default.yml:103

The convert processor for json.ClientIPASN (lines 103-117) no longer has the if: ctx.json?.ClientIPASN != '' condition that was present before. If ClientIPASN is an empty string (which may occur when the client IP doesn't have a known ASN), the conversion to long will fail and trigger the on_failure handler. This appends an error message to error.message, which then causes the event.kind to be incorrectly set to pipeline_error at line 186 (if: ctx.error?.message != null). This means valid NEL report events with empty ASN fields will be incorrectly categorized as pipeline errors. Note that the Timestamp convert processor at lines 48-53 correctly includes the empty string check (ctx.json.Timestamp != ''), suggesting this omission was unintentional.

🤖 Copy this AI Prompt to have your agent fix this: 
In file packages/cloudflare_logpush/data_stream/firewall_event/elasticsearch/ingest_pipeline/default.yml around line 135:

The `convert` processors for `json.EdgeResponseStatus`, `json.ClientASN`, `json.MatchIndex`, and `json.OriginResponseStatus` attempt to convert empty strings to `long`, which fails and triggers `pipeline_error` in `event.kind` even though these are valid optional fields. Previously these had `if: ctx.json?.field != ''` guards that were removed. Consider restoring the empty string checks to prevent valid documents with empty optional fields from being marked as errors.

Also found in 1 other location(s):
- packages/cloudflare_logpush/data_stream/nel_report/elasticsearch/ingest_pipeline/default.yml:103 -- The `convert` processor for `json.ClientIPASN` (lines 103-117) no longer has the `if: ctx.json?.ClientIPASN != ''` condition that was present before. If `ClientIPASN` is an empty string (which may occur when the client IP doesn't have a known ASN), the conversion to `long` will fail and trigger the `on_failure` handler. This appends an error message to `error.message`, which then causes the `event.kind` to be incorrectly set to `pipeline_error` at line 186 (`if: ctx.error?.message != null`). This means valid NEL report events with empty ASN fields will be incorrectly categorized as pipeline errors. Note that the `Timestamp` convert processor at lines 48-53 correctly includes the empty string check (`ctx.json.Timestamp != ''`), suggesting this omission was unintentional.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 12, 2026

💚 Build Succeeded

History

cc @brijesh-elastic

efd6
efd6 reviewed Apr 12, 2026
View reviewed changes
...ages/cloudflare_logpush/data_stream/access_request/elasticsearch/ingest_pipeline/default.yml Outdated
Comment on lines +70 to +71
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
Copy link
Copy Markdown
Contributor

@efd6 efd6 Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look right. Also below.

Copy link
Copy Markdown
Collaborator Author

@brijesh-elastic brijesh-elastic Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Cursor did that. It also reverting those changes in a33558a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 left review comments

@macroscopeapp macroscopeapp[bot] macroscopeapp[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@brijesh-elastic brijesh-elastic

Labels

Category: Integration quality Category: Quality used for SI planning documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:cloudflare_logpush Cloudflare Logpush Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

cloudflare_logpush: Normalise event.severity

3 participants

@brijesh-elastic @elasticmachine @efd6