Send CF Logs to New Relic and S3 #2961
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Minimum allowed coverage is Generated by 🐒 cobertura-action against 25c6a1e |
|
Per review with jadud, dan, tadhg, is there some way that we can get, via api call or some other means that show us the fluentbit heartbeat? Use case is that we want to know that the logdrain was successfully deployed after everything. (This really falls under the umbrella category of enhancing our testing suite and having api based smoke tests in our envs |
This was
linked to
issues
Dec 14, 2023
asteel-gsa
force-pushed
the
logshipper
branch
from
ba1431c
to
bee1e6f
Compare
asteel-gsa
force-pushed
the
logshipper
branch
from
bee1e6f
to
987acaa
Compare
|
@mogul any chance you are available for review? I would like to move this into production so we can get the credentials over to SoC and satisfy that poam |
mogul
previously requested changes
Dec 20, 2023
|
|
||
| # This has to happen after an application deployment because the manifest (currently) is responsible | ||
| # for binding the "logdrain service" to the "gsa-fac application". This also needs to be done | ||
| # based on the suspicion that fluentbit cannot register the incoming logs when it is initially |
There was a problem hiding this comment.
I'd still like to know what's going on here, but I'm OK with leaving this in if it's reliably addressing the problem.
There was a problem hiding this comment.
As far as I can tell, yes, this is reliably addressing the issue. Once everything is configured and the app is deploying, I only ever saw it go to a 201 instead of the 502 after a restart, or something needed to minorly modify the module. Probably something to investigate, but, I suppose that is the beauty of being the first consumers of this 🤣
terraform/shared/modules/cg-logshipper/fluentbit_config/fluentbit.conf
Outdated
Show resolved
Hide resolved
asteel-gsa
force-pushed
the
logshipper
branch
from
3370423
to
953cf91
Compare
asteel-gsa
force-pushed
the
logshipper
branch
from
5b0be41
to
25c6a1e
Compare
|
Terraform plan for dev Plan: 8 to add, 1 to change, 0 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
# module.dev.cloudfoundry_network_policy.logshipper-network-policy will be created
+ resource "cloudfoundry_network_policy" "logshipper-network-policy" {
+ id = (known after apply)
+ policy {
+ destination_app = "a6eb3870-2769-4aa9-a8f2-f1a31939f563"
+ port = "61443"
+ protocol = "tcp"
+ source_app = (known after apply)
}
}
# module.dev.cloudfoundry_user_provided_service.credentials will be updated in-place
~ resource "cloudfoundry_user_provided_service" "credentials" {
~ credentials = (sensitive value)
id = "03df74b7-065a-46df-9a85-0bac201bf36c"
name = "newrelic-creds"
~ tags = [
+ "newrelic-creds",
]
# (1 unchanged attribute hidden)
}
# module.dev.module.cg-logshipper.cloudfoundry_app.cg_logshipper_app will be created
+ resource "cloudfoundry_app" "cg_logshipper_app" {
+ buildpack = (known after apply)
+ buildpacks = [
+ "https://github.com/cloudfoundry/apt-buildpack",
+ "nginx_buildpack",
]
+ disk_quota = 256
+ enable_ssh = (known after apply)
+ environment = (sensitive value)
+ health_check_http_endpoint = (known after apply)
+ health_check_invocation_timeout = (known after apply)
+ health_check_timeout = (known after apply)
+ health_check_type = "process"
+ id = (known after apply)
+ id_bg = (known after apply)
+ instances = 1
+ memory = 256
+ name = "logshipper"
+ path = "../shared/modules/cg-logshipper/logshipper.zip"
+ ports = (known after apply)
+ source_code_hash = "d0a1938f9e105317e885d5e92d327e9845916cd7dd4432c08afa2e223c530c14"
+ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+ stack = (known after apply)
+ stopped = false
+ strategy = "rolling"
+ timeout = 180
+ routes {
+ port = (known after apply)
+ route = (known after apply)
}
+ service_binding {
+ service_instance = "03df74b7-065a-46df-9a85-0bac201bf36c"
}
+ service_binding {
+ service_instance = (known after apply)
}
+ service_binding {
+ service_instance = (known after apply)
}
}
# module.dev.module.cg-logshipper.cloudfoundry_route.logshipper will be created
+ resource "cloudfoundry_route" "logshipper" {
+ domain = "50ba3f69-cd54-4963-9172-14f3334b479e"
+ endpoint = (known after apply)
+ hostname = "fac-dev-logshipper"
+ id = (known after apply)
+ port = (known after apply)
+ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
}
# module.dev.module.cg-logshipper.cloudfoundry_user_provided_service.logdrain_service will be created
+ resource "cloudfoundry_user_provided_service" "logdrain_service" {
+ id = (known after apply)
+ name = "fac-logdrain"
+ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+ syslog_drain_url = (sensitive value)
}
# module.dev.module.cg-logshipper.cloudfoundry_user_provided_service.logshipper_creds will be created
+ resource "cloudfoundry_user_provided_service" "logshipper_creds" {
+ credentials = (sensitive value)
+ id = (known after apply)
+ name = "cg-logshipper-creds"
+ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+ tags = [
+ "logshipper-creds",
]
}
# module.dev.module.cg-logshipper.random_password.password will be created
+ resource "random_password" "password" {
+ bcrypt_hash = (sensitive value)
+ id = (known after apply)
+ length = 16
+ lower = true
+ min_lower = 0
+ min_numeric = 0
+ min_special = 0
+ min_upper = 0
+ number = true
+ numeric = true
+ result = (sensitive value)
+ special = false
+ upper = true
}
# module.dev.module.cg-logshipper.random_uuid.username will be created
+ resource "random_uuid" "username" {
+ id = (known after apply)
+ result = (known after apply)
}
# module.dev.module.cg-logshipper.module.s3-logshipper-storage.cloudfoundry_service_instance.bucket will be created
+ resource "cloudfoundry_service_instance" "bucket" {
+ id = (known after apply)
+ name = "log-storage"
+ replace_on_params_change = false
+ replace_on_service_plan_change = false
+ service_plan = "021bb2a3-7e11-4fc2-b06b-d9f5938cd806"
+ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+ tags = [
+ "logshipper-s3",
]
}
Plan: 8 to add, 1 to change, 0 to destroy.
Warning: Argument is deprecated
with module.dev.module.cg-logshipper.module.s3-logshipper-storage.cloudfoundry_service_instance.bucket,
on /tmp/terraform-data-dir/modules/dev.cg-logshipper.s3-logshipper-storage/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
14: recursive_delete = var.recursive_delete
Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases
(and 5 more similar warnings elsewhere)❌ Plan not applied in Deploy to Development and Management Environment #436 (Plan has changed) |
|
Terraform plan for meta No changes. Your infrastructure matches the configuration.✅ Plan applied in Deploy to Development and Management Environment #436 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue: #1019
Sketch:
flowchart TD A(S3 Bucket) -->|Bind Service| B{Logshipper App} D(New Relic Credentials) -->|Bind Service| B C(Logshipper Credentials) -->|Bind Service| B B -->|Send Logs| F(S3 Bucket) B -->|Send Logs| G(New Relic) E -->|Send Logs| B B --> H(Logdrain) H --> |Bind Service| E(gsa-fac Application)Key Components:
cg-logshipper-credswith a tag oflogshipper-credsis populated as a random user/pass generated via terraform that signifies the${HTTP_USER}AND${HTTP_PASS}credential set for thesyslog_drainnewrelic-credsis not a new credential service, but has now been given a tag ofnewrelic-credsand a content of the logs uri endpoint, in addition to the existing credential contentslog-storagewith a tag oflogshipper-s3is a dedicated s3 bucket that's sole purpose is the ingest and storage of allfluentbitlogs being sent. The credentials for this bucket, will be given to GSA SoC so that they may see the application metrics and logs.fac-logdrainis what is bound to thegsa-facapplication, so that it may stream logs to thecg-logshipperapplication. It consists of asyslog_drainuri.NOTE:
As of the present, the names for these services remain consistently named against the
.profile in the logshipper repountil PR #11 is merged, at which time, the.profilewill remove constraints on theservice-nameand instead be dependent on thetagsValidation:
newrelic.source:"api.logs".The
Fluent Bit - Heartbeatis a 60 second "heartbeat" that allows us to easily know if fluentbit is still operating. The empty spaces that are seen are infact application metrics and logs. The message is "empty", but the "raw_message" is not, so in order to see what that is, clicking the log will show the specifics, along with thetags.*of the "raw_message" that makes things relatively indexable.log-storages3 bucket allows us to view that logs are infact present from fluent bitA full path looks like
fluent-bit-logs/YYYY/MM/DD/HH/mm/object-chunk-nameTesting:
previewenvironment.