From 4dcf1a983a07f726025b371f8240e6ffe97b1ea8 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 15 Feb 2024 12:32:36 +0000 Subject: [PATCH 01/16] add scrubed keys --- security/privacy.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/security/privacy.md b/security/privacy.md index f4ef46ffc..a123425a2 100644 --- a/security/privacy.md +++ b/security/privacy.md @@ -29,9 +29,13 @@ The following keys are automatically scrubbed: - `apikey` - `access_token` - `auth` +- `authorizer` - `credentials` - `creds` - `sourceip` +- `clientip` +- `authorization` +- `x-api-key` Any nested field in your telemetry data that contains any of these automatically scrubbed keys will be blocked from ingestion by default. From c5e627e62b307b09a2a11a4b5cbc1ac2e3994618 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Fri, 16 Feb 2024 10:37:32 +0000 Subject: [PATCH 02/16] add refresh token to scrubbed --- security/privacy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/security/privacy.md b/security/privacy.md index a123425a2..0466a7492 100644 --- a/security/privacy.md +++ b/security/privacy.md @@ -36,6 +36,8 @@ The following keys are automatically scrubbed: - `clientip` - `authorization` - `x-api-key` +- `refresh_token` +- `refresh-token` Any nested field in your telemetry data that contains any of these automatically scrubbed keys will be blocked from ingestion by default. From c0ed0a5ca9bc872b063e03a9b3ec73d0e337fc84 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 22 Feb 2024 12:07:50 +0000 Subject: [PATCH 03/16] clarify otel for platforms --- sending-data/platforms/aws/aws-lambda/traces/index.md | 2 +- sending-data/platforms/opentelemetry.md | 11 +++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/sending-data/platforms/aws/aws-lambda/traces/index.md b/sending-data/platforms/aws/aws-lambda/traces/index.md index 5217e3611..b8e9a4a2c 100644 --- a/sending-data/platforms/aws/aws-lambda/traces/index.md +++ b/sending-data/platforms/aws/aws-lambda/traces/index.md @@ -67,7 +67,7 @@ Other observability tool layers and tags can adversely interact with the Baselim --- -## Remove OpenTelemetry +## Remove OpenTelemetry Instrumentation To remove the OpenTelemetry instrumentation from your AWS Lambda functions, remove the `baselime:tracing=true` tag from the function and Baselime will revert the function to un-instrumentate state. --- diff --git a/sending-data/platforms/opentelemetry.md b/sending-data/platforms/opentelemetry.md index c228785cd..90a989acd 100644 --- a/sending-data/platforms/opentelemetry.md +++ b/sending-data/platforms/opentelemetry.md @@ -24,8 +24,9 @@ Get your public Baselime API key in the [Baselime console](https://console.basel +++ --- -Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OTEL. -[!ref icon="../../assets/images/logos/aws.svg"](../private-link.md) +## Baselime Opentelemetry for Platforms +[!ref icon="../../assets/images/logos/logo_aws_lambda.png"](../platforms/aws/aws-lambda/traces/index.md) +[!ref icon="../../assets/images/logos/cloudflare.png"](../platforms/cloudflare/traces.md) --- @@ -35,3 +36,9 @@ Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OTE [!ref icon="../../assets/images/logos/python.svg"](../languages/python.md) [!ref icon="../../assets/images/logos/golang.svg"](../languages/go.md) +--- + +## AWS PrivateLink + +Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OpenTelemetry. +[!ref icon="../../assets/images/logos/aws.svg"](../private-link.md) From 742e0cdc62a27307484ad2f06e1e28c019b4688a Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 22 Feb 2024 12:12:58 +0000 Subject: [PATCH 04/16] change label --- sending-data/platforms/cloudflare/traces.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sending-data/platforms/cloudflare/traces.md b/sending-data/platforms/cloudflare/traces.md index 881b3b835..18282fc3f 100644 --- a/sending-data/platforms/cloudflare/traces.md +++ b/sending-data/platforms/cloudflare/traces.md @@ -1,6 +1,6 @@ --- order: 0 -label: Distributed Tracing +label: OpenTelemetry for Cloudflare Workers --- # OpenTelemetry for Cloudflare Workers From 079a3f28b159c72d6fc691ecb051257572739a87 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 22 Feb 2024 12:23:16 +0000 Subject: [PATCH 05/16] re-organise the opentelemetry section --- api/index.md | 2 +- sending-data/index.md | 2 +- .../{ => opentelemetry}/opentelemetry.md | 14 +++++------ .../opentelemetry}/private-link.md | 23 +++++++++++-------- 4 files changed, 22 insertions(+), 19 deletions(-) rename sending-data/platforms/{ => opentelemetry}/opentelemetry.md (62%) rename sending-data/{ => platforms/opentelemetry}/private-link.md (92%) diff --git a/api/index.md b/api/index.md index 9bd2d9c23..dff05316a 100644 --- a/api/index.md +++ b/api/index.md @@ -140,4 +140,4 @@ The response will return an array containing the list of environments the develo ### Send events to Baselime -Once you have obtained the public API for an environment, it can be used to send events to Baselime, either using the [Events API](../sending-data/events-api.md) or the [OpenTelemetry API](../sending-data/platforms/opentelemetry.md). \ No newline at end of file +Once you have obtained the public API for an environment, it can be used to send events to Baselime, either using the [Events API](../sending-data/events-api.md) or the [OpenTelemetry API](../sending-data/platforms/opentelemetry/opentelemetry.md). \ No newline at end of file diff --git a/sending-data/index.md b/sending-data/index.md index 4ca0d2e38..47ca3dec7 100644 --- a/sending-data/index.md +++ b/sending-data/index.md @@ -11,7 +11,7 @@ Baselime supports a variety of data sources, including logs, metrics, traces, an [!ref icon="../assets/images/logos/aws.svg"](./platforms/aws/index.md) [!ref icon="../assets/images/logos/vercel.png"](./platforms/vercel.md) [!ref icon="../assets/images/logos/cloudflare.png"](./platforms/cloudflare/index.md) -[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry.md) +[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry/opentelemetry.md) [!ref icon="../assets/images/logos/render.png"](./platforms/render.md) [!ref icon="../assets/images/logos/logo_docker_icon.png"](./platforms/docker.md) [!ref icon="../assets/images/logos/koyeb.png"](./platforms/koyeb.md) diff --git a/sending-data/platforms/opentelemetry.md b/sending-data/platforms/opentelemetry/opentelemetry.md similarity index 62% rename from sending-data/platforms/opentelemetry.md rename to sending-data/platforms/opentelemetry/opentelemetry.md index 90a989acd..900426a26 100644 --- a/sending-data/platforms/opentelemetry.md +++ b/sending-data/platforms/opentelemetry/opentelemetry.md @@ -25,20 +25,20 @@ Get your public Baselime API key in the [Baselime console](https://console.basel --- ## Baselime Opentelemetry for Platforms -[!ref icon="../../assets/images/logos/logo_aws_lambda.png"](../platforms/aws/aws-lambda/traces/index.md) -[!ref icon="../../assets/images/logos/cloudflare.png"](../platforms/cloudflare/traces.md) +[!ref icon="../../../assets/images/logos/logo_aws_lambda.png"](../../platforms/aws/aws-lambda/traces/index.md) +[!ref icon="../../../assets/images/logos/cloudflare.png"](../../platforms/cloudflare/traces.md) --- ## Baselime Opentelemetry Distros -[!ref icon="../../assets/images/logos/node.svg"](../languages/node.js.md) -[!ref icon="../../assets/images/logos/next.js.svg"](../languages/next.js.md) -[!ref icon="../../assets/images/logos/python.svg"](../languages/python.md) -[!ref icon="../../assets/images/logos/golang.svg"](../languages/go.md) +[!ref icon="../../../assets/images/logos/node.svg"](../../languages/node.js.md) +[!ref icon="../../../assets/images/logos/next.js.svg"](../../languages/next.js.md) +[!ref icon="../../../assets/images/logos/python.svg"](../../languages/python.md) +[!ref icon="../../../assets/images/logos/golang.svg"](../../languages/go.md) --- ## AWS PrivateLink Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OpenTelemetry. -[!ref icon="../../assets/images/logos/aws.svg"](../private-link.md) +[!ref icon="../../../assets/images/logos/aws.svg"](./private-link.md) diff --git a/sending-data/private-link.md b/sending-data/platforms/opentelemetry/private-link.md similarity index 92% rename from sending-data/private-link.md rename to sending-data/platforms/opentelemetry/private-link.md index 2218c7e4b..6984c322c 100644 --- a/sending-data/private-link.md +++ b/sending-data/platforms/opentelemetry/private-link.md @@ -7,13 +7,6 @@ order: -1 Baselime supports AWS PrivateLink, enabling you to securely send the data directly from your VPC to Baselime's platform without exposing your data to the public internet. -!!!info -Endpoint service name `com.amazonaws.vpce.eu-west-1.vpce-svc-03611009d136b2d65` -!!! -!!!warning -Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for both **HTTP** and **gRPC** when using PrivateLink -!!! - ## How to set it up +++ AWS Console * Navigate to "VPC" > "Endpoints" > "Create Endpoint" @@ -28,7 +21,8 @@ Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for bot * Once accepted navigate to "VPC" > "Endpoints" and select the endpoint you just created. * "Actions" > "Modify private DNS name" > Tick "Enable private DNS names" > "Save changes" -![Creating VPC Endpoint](../assets/images/illustrations/sending-data/privateLink/private_link_console_1.png) + +![Creating VPC Endpoint](../../../assets/images/illustrations/sending-data/privateLink/private_link_console_1.png) +++ CDK ```typescript @@ -110,15 +104,24 @@ resource "aws_vpc_endpoint" "baselime-otel" { ``` +++ +!!!info +Endpoint service name `com.amazonaws.vpce.eu-west-1.vpce-svc-03611009d136b2d65` +!!! +!!!warning +Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for both **HTTP** and **gRPC** when using PrivateLink +!!! + After about 60 seconds all your traffic to `otel-ingest.baselime.io` will be routed through the PrivateLink. +--- + ## How it works #### Without PrivateLink Endpoint When using Baselime without a PrivateLink endpoint, the DNS `otel-ingest.baslime.io` resolves to the public IP address of Baselime's platform. Your OTEL collector then sends the telemetry data to Baselime's platform over the public internet. Our endpoints are protected by TLS, so your data is encrypted in transit. -![Sending data without PrivateLink](../assets/images/illustrations/sending-data/privateLink/without_private_link.png) +![Sending data without PrivateLink](../../../assets/images/illustrations/sending-data/privateLink/without_private_link.png) #### With PrivateLink Endpoint When using Baselime with a PrivateLink endpoint, the DNS `otel-ingest.baslime.io` resolves the the private IP of a Network @@ -126,6 +129,6 @@ Interface that exists in your VPC, and the IP itself is one of from the CIDR ran then sends the data to Baselime's platform over the private network. This means that your data never leaves AWS infrastructure, and is never exposed to the public internet. -![Sending data with PrivateLink](../assets/images/illustrations/sending-data/privateLink/with_private_link.png) +![Sending data with PrivateLink](../../../assets/images/illustrations/sending-data/privateLink/with_private_link.png) Read more about [AWS PrivateLink](https://aws.amazon.com/privatelink/) on the AWS website. From 55bf5cab094484a52413c95a5c08c0bf342d09b8 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 22 Feb 2024 15:08:38 +0000 Subject: [PATCH 06/16] add service to logs enrichment --- sending-data/logs-enrichment.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/sending-data/logs-enrichment.md b/sending-data/logs-enrichment.md index 5cb701783..d7847b534 100644 --- a/sending-data/logs-enrichment.md +++ b/sending-data/logs-enrichment.md @@ -15,11 +15,12 @@ The available fields are: - `traceId` - `error` - `namespace` +- `service` You can add those fields to your logs to enable the [Requests](../analysing-data/overview.md) view in Baselime. !!! -Baselime automatically adds those fields to logs coming from cloud services with deep integrations such as [AWS Lambda](./platforms/aws/aws-lambda/index.md) and [Vercel](./platforms/vercel.md). +Baselime automatically adds those fields to logs coming from cloud services with deep integrations such as [AWS Lambda](./platforms/aws/aws-lambda/index.md), [Vercel](./platforms/vercel.md) and [Cloudflare Workers](./platforms/cloudflare/index.md). !!! --- @@ -116,7 +117,7 @@ function willThrow() { --- -## Grouping error by namespace or path +## Grouping logs by namespace or path Baselime enables you to group your logs by namespace or path by adding a `namespace` field to at least one log from a given request. @@ -143,6 +144,28 @@ app.get('/example', (req, res) => { }); ``` +--- + +## Grouping logs by service + +Baselime enables you to group your logs by service by adding a `service` field to all logs from the same service. + +For example: + +```javascript # + +const SERVICE = "queue-processor"; + +function backgrounJob(data) { + console.log(JSON.stringify({ message: "Starting processing", service: SERVICE, namespace: data.id })) + + complexCalculation(); + + console.log(JSON.stringify({ message: "Processing completed", service: SERVICE, namespace: data.id })) +} +``` + + --- ## Correlating logs and traces From 18079d610906b9b9901875c8e5ea1f9b99f93087 Mon Sep 17 00:00:00 2001 From: walmsles <2704782+walmsles@users.noreply.github.com> Date: Sun, 25 Feb 2024 21:45:54 +1100 Subject: [PATCH 07/16] fix(typo): fix spelling of or to our in last paragraph --- what-is-baselime.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/what-is-baselime.md b/what-is-baselime.md index 1f251c23a..792fc7e51 100644 --- a/what-is-baselime.md +++ b/what-is-baselime.md @@ -49,5 +49,5 @@ Troubleshoot infrastructure and application issues with high cardinality data an Query against any nested field and automatically surface anomalies fast; regardless of how unusual or unique this state of your application is. ### Take control of your data and costs -Use or backend or **Bring Your Own Backend**. Up to 6x more value than incumbents. No per-function pricing, no per-seat pricing, no per-alert pricing. Start at $0 and scale up as your applications grow, with no hidden fees. +Use our backend or **Bring Your Own Backend**. Up to 6x more value than incumbents. No per-function pricing, no per-seat pricing, no per-alert pricing. Start at $0 and scale up as your applications grow, with no hidden fees. From 77ebc7daf37c2cfa9ca4da10f0c6d843ced6b8db Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Tue, 27 Feb 2024 17:29:28 +0000 Subject: [PATCH 08/16] * Adds fly.io docs * Updates OTEL docs copy to include metrics and logs --- sending-data/platforms/flyio.md | 31 +++++++++++++++++++ .../platforms/opentelemetry/opentelemetry.md | 12 +++---- 2 files changed, 37 insertions(+), 6 deletions(-) create mode 100644 sending-data/platforms/flyio.md diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md new file mode 100644 index 000000000..5cebeb7a1 --- /dev/null +++ b/sending-data/platforms/flyio.md @@ -0,0 +1,31 @@ +--- +label: Fly.io +order: -5 +--- + +# Fly.io +[Fly.io](https://fly.io) is a platform for running applications globally. You can stream your Fly.io services logs to Baselime using their log exporter. + +--- + +## Obtain Baselime API Key +Get your `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io). + +## Create a Logger service and set the secrets using following commands: +1. `fly launch --image flyio/log-shipper:latest` +2. `fly secrets set ACCESS_TOKEN=$(fly auth token)` +3. `fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio` +4. `fly secrets set HTTP_TOKEN=BASELIME_API_KEY` + +!!!info +You can change the destination "dataset" in Baselime.io by changing ~~`flyio`~~ part of `https://events.baselime.io/v1/flyio` +!!! + +## Navigate to Baselime console + +You can now deploy the logger service to Fly.io and start streaming your logs to [Baselime console](https://console.baselime.io). +Once the Log Stream is created, all your logs from Fly.io will be available in Baselime for searching, queries, dashboard and alerts. + +[!ref](../events-validation.md) + +For more configuration options of the logger consult [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration) documentation. \ No newline at end of file diff --git a/sending-data/platforms/opentelemetry/opentelemetry.md b/sending-data/platforms/opentelemetry/opentelemetry.md index 900426a26..5b3da8eb2 100644 --- a/sending-data/platforms/opentelemetry/opentelemetry.md +++ b/sending-data/platforms/opentelemetry/opentelemetry.md @@ -3,25 +3,22 @@ label: OpenTelemetry order: -4 --- -If your codebase is already instrumented with [OpenTelemetry](https://opentelemetry.io/), you can start sending your tracing data to Baselime today. - +If your codebase is already instrumented with [OpenTelemetry](https://opentelemetry.io/), you can start sending your traces, metrics and logs to Baselime today. Add the Baselime OpenTelemetry endpoint to your exporter: +++HTTP - Endpoint `otel.baselime.io/v1/` - Header: `x-api-key: ` - Header: `x-baselime-dataset: ` - the dataset to send the data to -!!! -Get your public Baselime API key in the [Baselime console](https://console.baselime.io). -!!! +++gRPC - Endpoint `otel-ingest.baselime.io:4317` - Header: `api-key: ` - Header: `io.baselime.dataset: ` - the dataset to send the data to ++++ + !!! Get your public Baselime API key in the [Baselime console](https://console.baselime.io). !!! -+++ --- ## Baselime Opentelemetry for Platforms @@ -42,3 +39,6 @@ Get your public Baselime API key in the [Baselime console](https://console.basel Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OpenTelemetry. [!ref icon="../../../assets/images/logos/aws.svg"](./private-link.md) + +## Language APIs and SDKs +Consult the [OTEL documentation](https://opentelemetry.io/docs/languages/) to find support for traces, metrics and logs for your languages \ No newline at end of file From 40a3035fb390cffd0742f677421b2d217527a659 Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Tue, 27 Feb 2024 18:03:41 +0000 Subject: [PATCH 09/16] Removes data-validation reference from flyio docs --- sending-data/platforms/flyio.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md index 5cebeb7a1..ff8ea6a09 100644 --- a/sending-data/platforms/flyio.md +++ b/sending-data/platforms/flyio.md @@ -26,6 +26,5 @@ You can change the destination "dataset" in Baselime.io by changing ~~`flyio`~~ You can now deploy the logger service to Fly.io and start streaming your logs to [Baselime console](https://console.baselime.io). Once the Log Stream is created, all your logs from Fly.io will be available in Baselime for searching, queries, dashboard and alerts. -[!ref](../events-validation.md) +For more configuration options of the logger consult [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration) documentation. -For more configuration options of the logger consult [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration) documentation. \ No newline at end of file From 8aa0ebe078a712b5de907fa4797dbe176d4843ab Mon Sep 17 00:00:00 2001 From: Nico Date: Thu, 29 Feb 2024 16:10:40 +0100 Subject: [PATCH 10/16] Create logo_fly_icon.png --- assets/images/logos/logo_fly_icon.png | Bin 0 -> 3619 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 assets/images/logos/logo_fly_icon.png diff --git a/assets/images/logos/logo_fly_icon.png b/assets/images/logos/logo_fly_icon.png new file mode 100644 index 0000000000000000000000000000000000000000..af60feb1b41c301d5341aaf6547a2452c2d8e52d GIT binary patch literal 3619 zcmV+;4&3pHP)Px#32;bRa{vG#H~;`4H~~eLF<<}y0G3cpR7FQ{OeGgx zWI@4jR_Jk8=vYF%BoHq)#a8~GXSLk6~osdtB+DZtbRV?v7>al4t9!b?mTByBSmgnw7asiVHi$JmU8sc>hMLcZ)$0000HbW%=J z0Iv{b<{;sLKY+lefNn#OFqZTx>i_@@`AI}URCwC$oLOVyxDtg;2NH&C0~i})z?j7g z*g*dOpBr0iRY`IneLH<0qIu~_&yX*tPL)bqJnv`xjGyuUJ$}8t_Ii(x<7PZQ_MTqf z{wmGeQ*UgH-qXK-Nt;&-4IhuMgYU{X-q^EDh8GMSvkOqHCp6q%# z<>JrRk4*o?>-X?H5hbpt$z&pd#^=f9L;cmagpKCbc`|`O?Z2NszLDt3$^(G1RxpM1 z^zHGQP@r+nzHt5?qBp`*PD~&nn)LbGx90oF29%v3(c`agH1x;B1|;Kn%KmiF24j~% z7$_58y!`2)^;`}jK;w(}>Xpy`-zL!61tfD%*yw*D(XYYK7(I8y2v9D+%#*)oe%4*{ z-06AuH<&p(R7pRiOY(|LYfg;*GJN@*~-<$eyftnAR8y(oh4WwKxVm9SOj47C6 zjW(MXdtSBjIP_^hrlvf32aty4#j#$^43PklO{78J!V}1FHKsH?U3LMTu!)M*YBgIa zBxNK9)FAZ-H}S~)V#5IXxCw~=O$*dAAtI?!`wb}(=h^S}8VJU#iT+(K}kQ}Mw(~~(qEP%EzvJ;NPvVu$j}^UTXncPxBm^dbQ!gSXauJxt@?fMI6O3BwAx4SRebI?xWi-T9+Wq6Y`PWwze-(ie+&uC1Oar zWY2iipo{t*@5+#)rvVC>)F+DK!kPHF?aR3ks3k%gk=zFgEj%rd;ULP>5LK?2}h(c@#xM`W{+$bAH~cLC`PlL9qFr6bSG zEj>{aOW*Gxni)a!aLxzZL58C_P4k&#ip^|5ClIjg*Z>Kphytx?qCp^sK+r)^`>BP! zHs&;9B4D-Oj#!F>tC4L=^U$;kQhr8xTFKiFt*o}r?K|n!a_Ix4$dYMFqPYgz!C50f z0>NF7#Ac3& z`4TA3h(x){d2ybQ`Dv;pT7Dvt?$f-Eq&F=saGC+DQK^B>+R#ao0*!#LrHi)2*4#u+ zAg4g>$2Z@l#uDX1AbCpG@JJ$S&>~risEJ%aYfBOGwLr@>Es#W<<9rh4Dw0GIfuO14 z7)JKQ7eEU*(?d_I#ta#VDU>7ylqlx{eMp{z1~L_jp^4VmMBZ*AyXg)K@sI(ab|fW` zhyk4?564jw5hRdjTA>*NQVQ8E#5FQW)HGzF zS%y-q5a^9cGh&IUkU70tJ*2X11q&lfvdHu_uS4xWsCz8z3K3HSh3I4!cpIxi&_h6> zrKu2E$rcBa_6dv_obOE!Sxr5gy1F5vkVvv4EsWN@Hx?h(kSU;rb?R`N8e~}_Gweg& z@yfA6sScShO$4(H7DlsD^LA*3vMiJyQh5*4zShfCG1ZT2N5tUlDv(^0tKM9+d#FR9 z3Ty+5Y^xN{!7yAZ19wC;>!pX(ri`=wa5QLC zA!X|$+Hkl6TPJ94CH)XscaTjY*r2%<9bs%Q)eX?0MtgW?1*E!*LNbQmEk>1M7z@9oo!T z2Dw@^k!9J(<$TRlmL+N)OYout2o#|$eZ7fh0$DddVp@rWMzz^f*GXcj(zM5SMGOK3 zmMv;ydL^|e&5fQq!HY6Dv6B4{#iPqm(7D;5fI_ z0QQIG*8L&J_K=b&O=wS@K>Gj(Eo}mAjio)>urj7sMil8T*0OrV8dWCG(z>#DZ=rd+ zPxoWO$hvyfsaA_zt7;N0ysSSKMNwN;_IMlCvm7kUaBYl2Wuh3_T58m+^6R=j`iUN< zK*vgGXx%zDc3~ZPCTnA8Zt8(Kgv}xhg8!4Ih=%jXz-Bcp)5dYR+&EuYO5=*0mBnW2 zi>L{Iv$a~P!?gqgX`^Lp>)8SU>ONX9T%;sYB57TKeYAmK0WGXsp7Pw))5m3VyREg0 zIMNYov`h^o6fRz&;Wly=InJlVS%)QNQyMe9qHesI6;z~UO2!h&Lyku>*Rmb~S@Vn# zH4@e0vI{JbSStQUqz?hgX$=B7hJhC7;u$`2LAQO>V9FCDL0gpN>gR8g7UKG)p zD?PTK*?nDrDT3s`jGYCi-SXJNc7Pm9^NXZ&4P?a3cD~R8p%dxC_YIZsKu3xCXDLm6 zz!@+;CI_oCu{;AVqBpE0k>81<+SWqzxB>=&j7HWtxj!16Pj&PiOuoj^ZFJH8Kvh8x zIdmt)FYI{w!k0Pa>!7Xe`>p!>`iugAaPz1G@|Gz)uEU64*rC_H!0i~4Xy`IQo!K7! zWwqZiVsc)OkI9T_l?%kL>vUddYHd_xj(3Pf2alG9Txzg5ND8#*^J|p`5V%gz02S_+ zktN&5ln}=2F=V8BYA8< zRs*${;@)ldD)F5i^F4>1NN09oQ3qd%TZ2o(NTAJwdGnOqTn_S2AG8VpT{o*O@-p-@ z-Mpt)n1`O%w@9QjtB4C5@O4m(?;Yjks{RLLBGPMSdGfZ~k++HuI@9*|+xa{(hyE=9#me@&O!*mQ zys6Q?yx-23iFI5N5qJ|AOo=`llc(`f(TwW;;)Bu)4`uCYI-8NVB?0n}^=;{(H~z}_ zaPmNF*H|8`&uH-FUoUctovqd$7gO?rQ~9U=(DCxN_qExggGgCBD^R$kKK+;O-&ioX zOQq~G`E*?Me_d>-KA+*Ze~Hfx)!Aun&;(aSt)|n(=zVze3-^EDUY|Z4FaO*)Uhx0c pj)#ZgpZw(#KjUZojGyuK@gIOY35YiU{51do002ovPDHLkV1nds&({C| literal 0 HcmV?d00001 From 814ebeb38348e7849d2652e7697da5e320dfd106 Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Thu, 29 Feb 2024 15:13:26 +0000 Subject: [PATCH 11/16] Adds fly.io to platforms page --- sending-data/index.md | 15 ++++++++------- sending-data/platforms/flyio.md | 3 +-- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/sending-data/index.md b/sending-data/index.md index 47ca3dec7..d3e04282c 100644 --- a/sending-data/index.md +++ b/sending-data/index.md @@ -9,23 +9,24 @@ Baselime supports a variety of data sources, including logs, metrics, traces, an ### Platforms [!ref icon="../assets/images/logos/aws.svg"](./platforms/aws/index.md) -[!ref icon="../assets/images/logos/vercel.png"](./platforms/vercel.md) [!ref icon="../assets/images/logos/cloudflare.png"](./platforms/cloudflare/index.md) -[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry/opentelemetry.md) -[!ref icon="../assets/images/logos/render.png"](./platforms/render.md) [!ref icon="../assets/images/logos/logo_docker_icon.png"](./platforms/docker.md) +[!ref icon="../assets/images/logos/logo_fly_icon.png"](./platforms/flyio.md) [!ref icon="../assets/images/logos/koyeb.png"](./platforms/koyeb.md) -[!ref icon="../assets/images/logos/vector.png"](./platforms/vector.md) [!ref icon="../assets/images/logos/logo_kubernetes_icon.png"](./platforms/kubernetes.md) +[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry/opentelemetry.md) +[!ref icon="../assets/images/logos/render.png"](./platforms/render.md) +[!ref icon="../assets/images/logos/vector.png"](./platforms/vector.md) +[!ref icon="../assets/images/logos/vercel.png"](./platforms/vercel.md) ### Languages -[!ref icon="../assets/images/logos/node.svg"](./languages/node.js.md) -[!ref icon="../assets/images/logos/python.svg"](./languages/python.md) [!ref icon="../assets/images/logos/golang.svg"](./languages/go.md) [!ref icon="../assets/images/logos/next.js.svg"](./languages/next.js.md) +[!ref icon="../assets/images/logos/node.svg"](./languages/node.js.md) +[!ref icon="../assets/images/logos/pino.png"](./languages/pino.js.md) +[!ref icon="../assets/images/logos/python.svg"](./languages/python.md) [!ref icon="../assets/images/logos/svelte.svg"](./languages/sveltekit.md) [!ref icon="../assets/images/logos/logo_react_icon.png"](./languages/react.md) -[!ref icon="../assets/images/logos/pino.png"](./languages/pino.js.md) [!ref icon="../assets/images/logos/winston.png"](./languages/winston.md) ### Direct Sources diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md index ff8ea6a09..f7649142c 100644 --- a/sending-data/platforms/flyio.md +++ b/sending-data/platforms/flyio.md @@ -23,8 +23,7 @@ You can change the destination "dataset" in Baselime.io by changing ~~`flyio`~~ ## Navigate to Baselime console -You can now deploy the logger service to Fly.io and start streaming your logs to [Baselime console](https://console.baselime.io). -Once the Log Stream is created, all your logs from Fly.io will be available in Baselime for searching, queries, dashboard and alerts. +You can now navigate to [Baselime console](https://console.baselime.io) and see your logs from Fly.io services. For more configuration options of the logger consult [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration) documentation. From 17cd5a2c6df27865786e6878ec092c96459a0ce2 Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Thu, 29 Feb 2024 16:01:55 +0000 Subject: [PATCH 12/16] edits to fly.io receptor docs --- sending-data/platforms/flyio.md | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md index f7649142c..ba14a46ea 100644 --- a/sending-data/platforms/flyio.md +++ b/sending-data/platforms/flyio.md @@ -4,26 +4,23 @@ order: -5 --- # Fly.io -[Fly.io](https://fly.io) is a platform for running applications globally. You can stream your Fly.io services logs to Baselime using their log exporter. ---- +[Fly.io](https://fly.io) is a platform for running applications globally. You can stream your Fly.io service logs to Baselime using the [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration). -## Obtain Baselime API Key -Get your `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io). +--- -## Create a Logger service and set the secrets using following commands: -1. `fly launch --image flyio/log-shipper:latest` -2. `fly secrets set ACCESS_TOKEN=$(fly auth token)` -3. `fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio` -4. `fly secrets set HTTP_TOKEN=BASELIME_API_KEY` +## Setup -!!!info -You can change the destination "dataset" in Baselime.io by changing ~~`flyio`~~ part of `https://events.baselime.io/v1/flyio` -!!! +**Step 1:** Get your `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io). -## Navigate to Baselime console +**Step 2:** Create a Fly Log Shipper service and set the secrets using following commands: -You can now navigate to [Baselime console](https://console.baselime.io) and see your logs from Fly.io services. +```bash # +fly launch --image flyio/log-shipper:latest` +fly secrets set ACCESS_TOKEN=$(fly auth token)` +fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio` +fly secrets set HTTP_TOKEN=BASELIME_API_KEY` +``` -For more configuration options of the logger consult [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration) documentation. +Once these steps are completed, logs from all services deployed on Fly.io will be streamed to Baselime, and available for search, queries, alerts and dashboards. From 9cabf3b0f647bb39a7f07f49ddd80abc3c47d64c Mon Sep 17 00:00:00 2001 From: Boris Tane Date: Fri, 1 Mar 2024 07:00:57 +0000 Subject: [PATCH 13/16] add sampling section --- sending-data/index.md | 1 + sending-data/sampling.md | 55 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 sending-data/sampling.md diff --git a/sending-data/index.md b/sending-data/index.md index d3e04282c..69d0f5531 100644 --- a/sending-data/index.md +++ b/sending-data/index.md @@ -41,3 +41,4 @@ Baselime supports a variety of data sources, including logs, metrics, traces, an [!ref Correlating logs with traces](./logs-correlation.md) [!ref Enriching Logs for faster debugging](./logs-enrichment.md) [!ref Data Validation](./events-validation.md) +[!ref Sampling](./sampling.md) diff --git a/sending-data/sampling.md b/sending-data/sampling.md new file mode 100644 index 000000000..eb7a4b12c --- /dev/null +++ b/sending-data/sampling.md @@ -0,0 +1,55 @@ +--- +label: Sampling +order: -11 +--- + +Sampling enables you to control the volume of data processed and ingested in Baselime, by selecting only a portion of your data, whilst keeping the ability to extrapolate key insights from the selected set. + +--- + +## When should you consider sampling? + +You should consider sampling when your data volumes are high, when you exceed multiple billion events per day, or **multiple thousand requests per second**. + +Sampling will enable you to reduce the overall volume of data processed and stored, and keep only data where "interesting things" occur. Interesting things could be errors, events with specific attributes, or events with high latencies associated. + +--- + +## Automatic tail sampling with Baselime + +!!! +Automatic tail sampling is available only on our Enterprise Tier +!!! + +You can enable automatic tail sampling on Baselime for each of your datasets. + +With tail sampling, the decision to sample a trace or request happens at the end of the processing, by considering all of the spans and events within the trace or request. A trace/request is sampled (ingested) when: + +- At least one event has an error (for logs, log levels are considered) +- At least one event has a status code > 399 +- it meets the deterministic probability sampling, based on a fixed sampling rate + +If any event (logs, spans and span events) associated with a trace or request meets the criteria, the entire trace or request is sampled. Otherwise, it is dropped. + +If your events are not associated to any trace or request, they are sampled using the same criteria, but are kept individually instead of as part of a request. + +!!! +Learn how to [enrich your logs with request IDs](./logs-enrichment.md) for faster troubleshooting. +!!! + +--- + +## How Baselime works with sampled data + +When Baselime automatically samples your data, it also saves a `$baselime.samplingInterval` on each of your events. If you decide to sample with a `N` sampling rate, Baselime will automatically store `1/N` as the `$baselime.samplingInterval`. + +Baselime then uses the sampling interval to compute aggregations such as `COUNT`, `SUM`, `AVG` and all percentiles. + +In other words, Baselime enables you to perfrom **accurate calculations** that are representative of the entire dataset, even with a sampled dataset. + +Baselime also surfaces the average sample rate in the query results in the [console](https://console.baselime.io). This gives you an idea of the sample rate of the events queried when running a query. + +### `COUNT_DISTINCT` + +`COUNT_DISTINCT` enables you to compute the number of unique values a property take in your dataset. This is impossible to accurately compute with sampled data. When performing a `COUNT_DISTINCT` aggregation, Baselime will count the number of unique values present in the sampled dataset and will not compensate for the sampling rate. + From d37544bbd82db8ed8dddf2028c0adb77eff782c1 Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Mon, 4 Mar 2024 14:13:31 +0000 Subject: [PATCH 14/16] * Removes backticks from fly.io shell commands * Adds reference to OTEL in fly.io --- sending-data/platforms/flyio.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md index ba14a46ea..47135fd6e 100644 --- a/sending-data/platforms/flyio.md +++ b/sending-data/platforms/flyio.md @@ -16,11 +16,12 @@ order: -5 **Step 2:** Create a Fly Log Shipper service and set the secrets using following commands: ```bash # -fly launch --image flyio/log-shipper:latest` -fly secrets set ACCESS_TOKEN=$(fly auth token)` -fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio` -fly secrets set HTTP_TOKEN=BASELIME_API_KEY` +fly launch --image flyio/log-shipper:latest +fly secrets set ACCESS_TOKEN=$(fly auth token) +fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio +fly secrets set HTTP_TOKEN=BASELIME_API_KEY ``` Once these steps are completed, logs from all services deployed on Fly.io will be streamed to Baselime, and available for search, queries, alerts and dashboards. +[!ref Instrument your fly.io tracing and metrics using OpenTelemetry](../platforms/opentelemetry/opentelemetry.md) \ No newline at end of file From 4a2a76b19c1b8d95e0bcba1296fc1b136ecb9553 Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Mon, 4 Mar 2024 14:51:31 +0000 Subject: [PATCH 15/16] Adds reference to OTEL in fly.io doc --- sending-data/platforms/flyio.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sending-data/platforms/flyio.md b/sending-data/platforms/flyio.md index 47135fd6e..7bc11476b 100644 --- a/sending-data/platforms/flyio.md +++ b/sending-data/platforms/flyio.md @@ -24,4 +24,4 @@ fly secrets set HTTP_TOKEN=BASELIME_API_KEY Once these steps are completed, logs from all services deployed on Fly.io will be streamed to Baselime, and available for search, queries, alerts and dashboards. -[!ref Instrument your fly.io tracing and metrics using OpenTelemetry](../platforms/opentelemetry/opentelemetry.md) \ No newline at end of file +[!ref icon="../../assets/images/logos/logo_open_telemetry.png" text="Instrument fly.io tracing and metrics using OpenTelemetry"](../platforms/opentelemetry/opentelemetry.md) \ No newline at end of file From 09f8a3e9d3faebf5c1cd9491c3dab9739f591acc Mon Sep 17 00:00:00 2001 From: Maksym Makuch Date: Mon, 4 Mar 2024 14:51:36 +0000 Subject: [PATCH 16/16] Updates OTEL docs to remove difference in headers between http and gRPC --- .../platforms/opentelemetry/opentelemetry.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/sending-data/platforms/opentelemetry/opentelemetry.md b/sending-data/platforms/opentelemetry/opentelemetry.md index 5b3da8eb2..966f38bf4 100644 --- a/sending-data/platforms/opentelemetry/opentelemetry.md +++ b/sending-data/platforms/opentelemetry/opentelemetry.md @@ -5,21 +5,21 @@ order: -4 If your codebase is already instrumented with [OpenTelemetry](https://opentelemetry.io/), you can start sending your traces, metrics and logs to Baselime today. -Add the Baselime OpenTelemetry endpoint to your exporter: +**Step 1:** Get `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io). + +**Step 2:** Set the headers +```yaml +x-api-key: +x-baselime-dataset: +``` + +**Step 3:** Set exporter endpoint +++HTTP - Endpoint `otel.baselime.io/v1/` -- Header: `x-api-key: ` -- Header: `x-baselime-dataset: ` - the dataset to send the data to +++gRPC - Endpoint `otel-ingest.baselime.io:4317` -- Header: `api-key: ` -- Header: `io.baselime.dataset: ` - the dataset to send the data to +++ -!!! -Get your public Baselime API key in the [Baselime console](https://console.baselime.io). -!!! - --- ## Baselime Opentelemetry for Platforms [!ref icon="../../../assets/images/logos/logo_aws_lambda.png"](../../platforms/aws/aws-lambda/traces/index.md)