merge

api/index.md

@@ -140,4 +140,4 @@ The response will return an array containing the list of environments the develo
 
 ### Send events to Baselime
 
-Once you have obtained the public API for an environment, it can be used to send events to Baselime, either using the [Events API](../sending-data/events-api.md) or the [OpenTelemetry API](../sending-data/platforms/opentelemetry.md).
+Once you have obtained the public API for an environment, it can be used to send events to Baselime, either using the [Events API](../sending-data/events-api.md) or the [OpenTelemetry API](../sending-data/platforms/opentelemetry/opentelemetry.md).

security/privacy.md

@@ -29,9 +29,15 @@ The following keys are automatically scrubbed:
 - `apikey`
 - `access_token`
 - `auth`
+- `authorizer`
 - `credentials`
 - `creds`
 - `sourceip`
+- `clientip`
+- `authorization`
+- `x-api-key`
+- `refresh_token`
+- `refresh-token`
 
 Any nested field in your telemetry data that contains any of these automatically scrubbed keys will be blocked from ingestion by default.
 

sending-data/index.md

@@ -9,23 +9,24 @@ Baselime supports a variety of data sources, including logs, metrics, traces, an
 
 ### Platforms
 [!ref icon="../assets/images/logos/aws.svg"](./platforms/aws/index.md)
-[!ref icon="../assets/images/logos/vercel.png"](./platforms/vercel.md)
 [!ref icon="../assets/images/logos/cloudflare.png"](./platforms/cloudflare/index.md)
-[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry.md)
-[!ref icon="../assets/images/logos/render.png"](./platforms/render.md)
 [!ref icon="../assets/images/logos/logo_docker_icon.png"](./platforms/docker.md)
+[!ref icon="../assets/images/logos/logo_fly_icon.png"](./platforms/flyio.md)
 [!ref icon="../assets/images/logos/koyeb.png"](./platforms/koyeb.md)
-[!ref icon="../assets/images/logos/vector.png"](./platforms/vector.md)
 [!ref icon="../assets/images/logos/logo_kubernetes_icon.png"](./platforms/kubernetes.md)
+[!ref icon="../assets/images/logos/logo_open_telemetry.png"](platforms/opentelemetry/opentelemetry.md)
+[!ref icon="../assets/images/logos/render.png"](./platforms/render.md)
+[!ref icon="../assets/images/logos/vector.png"](./platforms/vector.md)
+[!ref icon="../assets/images/logos/vercel.png"](./platforms/vercel.md)
 
 ### Languages
-[!ref icon="../assets/images/logos/node.svg"](./languages/node.js.md)
-[!ref icon="../assets/images/logos/python.svg"](./languages/python.md)
 [!ref icon="../assets/images/logos/golang.svg"](./languages/go.md)
 [!ref icon="../assets/images/logos/next.js.svg"](./languages/next.js.md)
+[!ref icon="../assets/images/logos/node.svg"](./languages/node.js.md)
+[!ref icon="../assets/images/logos/pino.png"](./languages/pino.js.md)
+[!ref icon="../assets/images/logos/python.svg"](./languages/python.md)
 [!ref icon="../assets/images/logos/svelte.svg"](./languages/sveltekit.md)
 [!ref icon="../assets/images/logos/logo_react_icon.png"](./languages/react.md)
-[!ref icon="../assets/images/logos/pino.png"](./languages/pino.js.md)
 [!ref icon="../assets/images/logos/winston.png"](./languages/winston.md)
 
 ### Direct Sources
@@ -40,3 +41,4 @@ Baselime supports a variety of data sources, including logs, metrics, traces, an
 [!ref Correlating logs with traces](./logs-correlation.md)
 [!ref Enriching Logs for faster debugging](./logs-enrichment.md)
 [!ref Data Validation](./events-validation.md)
+[!ref Sampling](./sampling.md)

sending-data/logs-enrichment.md

@@ -15,11 +15,12 @@ The available fields are:
 - `traceId`
 - `error`
 - `namespace`
+- `service`
 
 You can add those fields to your logs to enable the [Requests](../analysing-data/overview.md) view in Baselime.
 
 !!!
-Baselime automatically adds those fields to logs coming from cloud services with deep integrations such as [AWS Lambda](./platforms/aws/aws-lambda/index.md) and [Vercel](./platforms/vercel.md).
+Baselime automatically adds those fields to logs coming from cloud services with deep integrations such as [AWS Lambda](./platforms/aws/aws-lambda/index.md), [Vercel](./platforms/vercel.md) and [Cloudflare Workers](./platforms/cloudflare/index.md).
 !!!
 
 ---
@@ -116,7 +117,7 @@ function willThrow() {
 
 ---
 
-## Grouping error by namespace or path
+## Grouping logs by namespace or path
 
 Baselime enables you to group your logs by namespace or path by adding a `namespace` field to at least one log from a given request.
 
@@ -143,6 +144,28 @@ app.get('/example', (req, res) => {
 });
 ```
 
+---
+
+## Grouping logs by service
+
+Baselime enables you to group your logs by service by adding a `service` field to all logs from the same service.
+
+For example:
+
+```javascript #
+
+const SERVICE = "queue-processor";
+
+function backgrounJob(data) {
+  console.log(JSON.stringify({ message: "Starting processing", service: SERVICE, namespace: data.id }))
+
+  complexCalculation();
+
+  console.log(JSON.stringify({ message: "Processing completed", service: SERVICE, namespace: data.id }))
+}
+```
+
+
 ---
 
 ## Correlating logs and traces

sending-data/platforms/aws/aws-lambda/traces/index.md

@@ -67,7 +67,7 @@ Other observability tool layers and tags can adversely interact with the Baselim
 
 ---
 
-## Remove OpenTelemetry
+## Remove OpenTelemetry Instrumentation
 To remove the OpenTelemetry instrumentation from your AWS Lambda functions, remove the `baselime:tracing=true` tag from the function and Baselime will revert the function to un-instrumentate state.
 
 ---

sending-data/platforms/cloudflare/traces.md

@@ -1,6 +1,6 @@
 ---
 order: 0
-label: Distributed Tracing
+label: OpenTelemetry for Cloudflare Workers
 ---
 
 # OpenTelemetry for Cloudflare Workers

sending-data/platforms/flyio.md

@@ -0,0 +1,27 @@
+---
+label: Fly.io
+order: -5
+---
+
+# Fly.io
+
+[Fly.io](https://fly.io) is a platform for running applications globally. You can stream your Fly.io service logs to Baselime using the [fly-log-shipper](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#provider-configuration).
+
+---
+
+## Setup
+
+**Step 1:** Get your `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io).
+
+**Step 2:** Create a Fly Log Shipper service and set the secrets using following commands:
+
+```bash #
+fly launch --image flyio/log-shipper:latest
+fly secrets set ACCESS_TOKEN=$(fly auth token)
+fly secrets set HTTP_URL=https://events.baselime.io/v1/flyio
+fly secrets set HTTP_TOKEN=BASELIME_API_KEY
+```
+
+Once these steps are completed, logs from all services deployed on Fly.io will be streamed to Baselime, and available for search, queries, alerts and dashboards.
+
+[!ref icon="../../assets/images/logos/logo_open_telemetry.png" text="Instrument fly.io tracing and metrics using OpenTelemetry"](../platforms/opentelemetry/opentelemetry.md)

sending-data/platforms/opentelemetry/opentelemetry.md

@@ -0,0 +1,44 @@
+---
+label: OpenTelemetry
+order: -4
+---
+
+If your codebase is already instrumented with [OpenTelemetry](https://opentelemetry.io/), you can start sending your traces, metrics and logs to Baselime today.
+
+**Step 1:** Get `BASELIME_API_KEY` from the [Baselime console](https://console.baselime.io).
+
+**Step 2:** Set the headers
+```yaml
+x-api-key: <BASELIME_API_KEY>
+x-baselime-dataset: <YOUR_DATASET>
+```
+
+**Step 3:** Set exporter endpoint
++++HTTP
+- Endpoint `otel.baselime.io/v1/`
++++gRPC
+- Endpoint `otel-ingest.baselime.io:4317`
++++
+
+---
+## Baselime Opentelemetry for Platforms
+[!ref icon="../../../assets/images/logos/logo_aws_lambda.png"](../../platforms/aws/aws-lambda/traces/index.md)
+[!ref icon="../../../assets/images/logos/cloudflare.png"](../../platforms/cloudflare/traces.md)
+
+---
+
+## Baselime Opentelemetry Distros
+[!ref icon="../../../assets/images/logos/node.svg"](../../languages/node.js.md)
+[!ref icon="../../../assets/images/logos/next.js.svg"](../../languages/next.js.md)
+[!ref icon="../../../assets/images/logos/python.svg"](../../languages/python.md)
+[!ref icon="../../../assets/images/logos/golang.svg"](../../languages/go.md)
+
+---
+
+## AWS PrivateLink
+
+Baselime supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) for OpenTelemetry.
+[!ref icon="../../../assets/images/logos/aws.svg"](./private-link.md)
+
+## Language APIs and SDKs
+Consult the [OTEL documentation](https://opentelemetry.io/docs/languages/) to find support for traces, metrics and logs for your languages

sending-data/private-link.md → sending-data/platforms/opentelemetry/private-link.md

@@ -7,13 +7,6 @@ order: -1
 
 Baselime supports AWS PrivateLink, enabling you to securely send the data directly from your VPC to Baselime's platform without exposing your data to the public internet.
 
-!!!info
-Endpoint service name `com.amazonaws.vpce.eu-west-1.vpce-svc-03611009d136b2d65`
-!!!
-!!!warning
-Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for both **HTTP** and **gRPC** when using PrivateLink
-!!!
-
 ## How to set it up
 +++ AWS Console
 * Navigate to "VPC" > "Endpoints" > "Create Endpoint"
@@ -28,7 +21,8 @@ Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for bot
 * Once accepted navigate to "VPC" > "Endpoints" and select the endpoint you just created.
   * "Actions" > "Modify private DNS name" > Tick "Enable private DNS names" > "Save changes"
 
-![Creating VPC Endpoint](../assets/images/illustrations/sending-data/privateLink/private_link_console_1.png)
+
+![Creating VPC Endpoint](../../../assets/images/illustrations/sending-data/privateLink/private_link_console_1.png)
 
 +++ CDK
 ```typescript
@@ -110,22 +104,31 @@ resource "aws_vpc_endpoint" "baselime-otel" {
 ```
 +++
 
+!!!info
+Endpoint service name `com.amazonaws.vpce.eu-west-1.vpce-svc-03611009d136b2d65`
+!!!
+!!!warning
+Use endpoint `otel-ingest.baselime.io` instead of ~~`otel.baselime.io`~~ for both **HTTP** and **gRPC** when using PrivateLink
+!!!
+
 After about 60 seconds all your traffic to `otel-ingest.baselime.io` will be routed through the PrivateLink.
 
+---
+
 ## How it works
 
 #### Without PrivateLink Endpoint
 When using Baselime without a PrivateLink endpoint, the DNS `otel-ingest.baslime.io` resolves to the public
 IP address of Baselime's platform. Your OTEL collector then sends the telemetry data to Baselime's platform
 over the public internet. Our endpoints are protected by TLS, so your data is encrypted in transit.
-![Sending data without PrivateLink](../assets/images/illustrations/sending-data/privateLink/without_private_link.png)
+![Sending data without PrivateLink](../../../assets/images/illustrations/sending-data/privateLink/without_private_link.png)
 
 #### With PrivateLink Endpoint
 When using Baselime with a PrivateLink endpoint, the DNS `otel-ingest.baslime.io` resolves the the private IP of a Network
 Interface that exists in your VPC, and the IP itself is one of from the CIDR range of your VPC. Your OTEL collector
 then sends the data to Baselime's platform over the private network. This means that your data never leaves AWS
 infrastructure, and is never exposed to the public internet.
 
-![Sending data with PrivateLink](../assets/images/illustrations/sending-data/privateLink/with_private_link.png)
+![Sending data with PrivateLink](../../../assets/images/illustrations/sending-data/privateLink/with_private_link.png)
 
 Read more about [AWS PrivateLink](https://aws.amazon.com/privatelink/) on the AWS website.

sending-data/sampling.md

@@ -1,7 +1,54 @@
-# Sampling
+---
+label: Sampling
+order: -11
+---
 
-Baselime has built in support sampling on our [enterprise plan](https://baselime.io/pricing) to enable maximum control over spend and data volumes. The sampling selects requests or traces to keep based on a percentage presence of the `$baselime.error` field. 
+Sampling enables you to control the volume of data processed and ingested in Baselime, by selecting only a portion of your data, whilst keeping the ability to extrapolate key insights from the selected set.
 
-## What is sampling
+---
 
-Sampling 
+## When should you consider sampling?
+
+You should consider sampling when your data volumes are high, when you exceed multiple billion events per day, or **multiple thousand requests per second**.
+
+Sampling will enable you to reduce the overall volume of data processed and stored, and keep only data where "interesting things" occur. Interesting things could be errors, events with specific attributes, or events with high latencies associated.
+
+---
+
+## Automatic tail sampling with Baselime
+
+!!!
+Automatic tail sampling is available only on our Enterprise Tier
+!!!
+
+You can enable automatic tail sampling on Baselime for each of your datasets.
+
+With tail sampling, the decision to sample a trace or request happens at the end of the processing, by considering all of the spans and events within the trace or request. A trace/request is sampled (ingested) when:
+
+- At least one event has an error (for logs, log levels are considered)
+- At least one event has a status code > 399
+- it meets the deterministic probability sampling, based on a fixed sampling rate
+
+If any event (logs, spans and span events) associated with a trace or request meets the criteria, the entire trace or request is sampled. Otherwise, it is dropped.
+
+If your events are not associated to any trace or request, they are sampled using the same criteria, but are kept individually instead of as part of a request.
+
+!!!
+Learn how to [enrich your logs with request IDs](./logs-enrichment.md) for faster troubleshooting.
+!!!
+
+---
+
+## How Baselime works with sampled data
+
+When Baselime automatically samples your data, it also saves a `$baselime.samplingInterval` on each of your events. If you decide to sample with a `N` sampling rate, Baselime will automatically store `1/N` as the `$baselime.samplingInterval`.
+
+Baselime then uses the sampling interval to compute aggregations such as `COUNT`, `SUM`, `AVG` and all percentiles.
+
+In other words, Baselime enables you to perfrom **accurate calculations** that are representative of the entire dataset, even with a sampled dataset.
+
+Baselime also surfaces the average sample rate in the query results in the [console](https://console.baselime.io). This gives you an idea of the sample rate of the events queried when running a query.
+
+### `COUNT_DISTINCT`
+
+`COUNT_DISTINCT` enables you to compute the number of unique values a property take in your dataset. This is impossible to accurately compute with sampled data. When performing a `COUNT_DISTINCT` aggregation, Baselime will count the number of unique values present in the sampled dataset and will not compensate for the sampling rate.

what-is-baselime.md

@@ -49,5 +49,5 @@ Troubleshoot infrastructure and application issues with high cardinality data an
 Query against any nested field and automatically surface anomalies fast; regardless of how unusual or unique this state of your application is.
 
 ### Take control of your data and costs
-Use or backend or **Bring Your Own Backend**. Up to 6x more value than incumbents. No per-function pricing, no per-seat pricing, no per-alert pricing. Start at $0 and scale up as your applications grow, with no hidden fees.
+Use our backend or **Bring Your Own Backend**. Up to 6x more value than incumbents. No per-function pricing, no per-seat pricing, no per-alert pricing. Start at $0 and scale up as your applications grow, with no hidden fees.