Skip to content

[File Delivery] Read from additional integration sources#6599

Open
pzl wants to merge 5 commits intoelastic:mainfrom
pzl:file-read-branching
Open

[File Delivery] Read from additional integration sources#6599
pzl wants to merge 5 commits intoelastic:mainfrom
pzl:file-read-branching

Conversation

@pzl
Copy link
Member

@pzl pzl commented Mar 17, 2026

What is the problem this PR solves?

Current file-delivery mechanism is for short-lived files (ILM enforced) that are targeted to specific agents (i.e. send this file to this agent).

This addresses the new need for sending files that are not agent-targeted, that may have different ILM settings (or no ILM at all)

How does this PR solve the problem?

  • Adds ?source query parameter to file API
  • when used, fleet server will locate the file using the alternately requested source
  • source indices follow a limited naming pattern to prevent arbitrary reads
  • naming pattern also uses product origin header to route to the integration. Currently limited to defend, as the only initial user

Background

For specific context, the features driving this are coming from Defend integration. On top of target file deliveries ("this file to this machine"), defend is adding long-lived widely available files to be repetitively delivered to any (or all). Users add files to a user-manageable "file library" of known files that can be delivered at will (tools, scripts, etc), and not designed to be cleared out by an ILM policy.

The management of these files is moved onto the user and integration, fleet server's role is only to read out those files following the naming conventions added here.

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

@pzl pzl added the enhancement New feature or request label Mar 17, 2026
@mergify
Copy link
Contributor

mergify bot commented Mar 17, 2026

This pull request does not have a backport label. Could you fix it @pzl? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@pzl pzl marked this pull request as ready for review March 18, 2026 14:52
@pzl pzl requested a review from a team as a code owner March 18, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant