Skip to content

enhancement(fluent source): Add support for forwarding over unix socket #22212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Apr 18, 2025

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented Jan 15, 2025

Summary

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

I couldn't work out how to get the integration tests working, but I tested by running fluent-bit with the following config

[SERVICE]
    Flush      5
    Daemon     off
    Log_Level  info

[INPUT]
    Name       cpu
    Tag        cpu_usage

[OUTPUT]
    Name          forward
    Match         *
    unix_path     /tmp/map/fluent.sock

And then running vector with

sources:
  fluent:
    type: fluent
    path: /tmp/map/fluent.sock

sinks:
  out:
    inputs:
      - fluent
    type: console
    encoding:
      codec: json

And seeing the metrics being transferred.

I think it should be a relatively straightforward lift for someone with familiarity with the existing integration testing harness to hook this up, but I got all sorts of errors.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

  • Please read our Vector contributor resources.
    • make check-all is a good command to run locally. This check is
      defined here. Some of these
      checks might not be relevant to your PR. For Rust changes, at the very least you should run:
      • cargo fmt --all
      • cargo clippy --workspace --all-targets -- -D warnings
      • cargo nextest run --workspace (alternatively, you can run cargo test --all)
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

@tustvold tustvold requested a review from a team as a code owner January 15, 2025 13:28
@bits-bot
Copy link

bits-bot commented Jan 15, 2025

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Jan 15, 2025
Copy link
Contributor Author

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately this change is largely mechanical and is just gluing together the existing decoder with the existing unix stream machinery, but there is definitely the possibility I have missed something subtle.

listen_path: PathBuf,
socket_file_mode: Option<u32>,
decoder: Decoder,
decoder: D,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is necessary to allow using the FluentDecoder. As the fluent protocol doesn't have an explicit framing mechanism that I could ascertain, there wasn't a good way to make this work with the codecs::Decoder setup which separates framing and decoding.

This, and better discoverability, is also the reason I opted not to implement this as a codec on the socket source, as suggested on the ticket

@pront pront self-assigned this Jan 23, 2025
@pront

This comment was marked as outdated.

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR @tustvold! Left a couple of review comments.

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tustvold, custom deserialization is always tricky.

@pront pront enabled auto-merge February 26, 2025 15:35
@tustvold
Copy link
Contributor Author

Is there some way I can help move this one along, I can't really effectively debug the CI (which appears to be failing for unrelated reasons) as it requires approval to run...

@pront
Copy link
Member

pront commented Mar 21, 2025

Is there some way I can help move this one along, I can't really effectively debug the CI (which appears to be failing for unrelated reasons) as it requires approval to run...

Yes, please add a changelog fragment. I re-enabled the checks.

auto-merge was automatically disabled March 22, 2025 09:18

Head branch was pushed to by a user without write access

@pront pront enabled auto-merge March 24, 2025 18:17
auto-merge was automatically disabled March 24, 2025 18:21

Head branch was pushed to by a user without write access

@pront pront enabled auto-merge March 24, 2025 18:22
auto-merge was automatically disabled April 16, 2025 14:29

Head branch was pushed to by a user without write access

@tustvold tustvold requested a review from a team as a code owner April 16, 2025 14:29
@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Apr 16, 2025
@tustvold
Copy link
Contributor Author

I've merged main and regenerated the docs, let me know if anything further is required

@pront pront enabled auto-merge April 16, 2025 19:36
@pront pront added this pull request to the merge queue Apr 16, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 16, 2025
@pront
Copy link
Member

pront commented Apr 18, 2025

Some windows related failures: https://github.com/vectordotdev/vector/actions/runs/14501794689/job/40683006134. We need one or more feature gates.

@tustvold
Copy link
Contributor Author

Hopefully fixed it, although I don't have a windows machine on which to test this...

@pront pront enabled auto-merge April 18, 2025 15:12
@pront pront added this pull request to the merge queue Apr 18, 2025
Merged via the queue into vectordotdev:master with commit 8057e63 Apr 18, 2025
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Source: fluent, allow sending events to unix socket
5 participants