-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[metricbeat] [gcp] add billing additional fields #36902
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
return createDetailedQuery(tableName, month, costType) | ||
} | ||
|
||
return createStandardQuery(tableName, month, costType) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets add an else if table doesn't match any pattern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The table pattern will always match the standard pattern even if user forgets to set it.
beats/x-pack/metricbeat/module/gcp/billing/billing.go
Lines 131 to 134 in b7af6f9
if m.config.TablePattern == "" { | |
m.logger.Warn("table_pattern is not set in config, \"gcp_billing_export_v1\" will be used by default.") | |
m.config.TablePattern = "gcp_billing_export_v1" | |
} |
b7af6f9
to
7c51bc8
Compare
events = append(events, createEvents(row, m.config.ProjectID)) | ||
case DetailedTableRowLen: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This casing would be better if done depending upon the type of table being queried rather than the number of rows ?
f61ae91
to
6a801bc
Compare
Standard document example:
{
"_index": ".ds-metricbeat-8.12.0-2023.10.19-000001",
"_id": "eh2USYsBN2wa5YEjwRKW",
"_score": 1,
"_source": {
"@timestamp": "2023-10-19T20:16:01.524Z",
"metricset": {
"name": "billing",
"period": 60000
},
"service": {
"type": "gcp"
},
"gcp": {
"billing": {
"service_description": "Compute Engine",
"invoice_month": "202310",
"project_id": "robust-catalyst-399814",
"billing_account_id": "011702-58A742-BEB4E7",
"total": 0,
"sku_id": "0D56-2F80-52A5",
"cost_type": "regular",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"tags": {},
"project_name": "My Project 70130"
}
},
"cloud.project.id": "robust-catalyst-399814",
"cloud.project.name": "My Project 70130",
"host": {
"os": {
"platform": "ubuntu",
"version": "22.04.2 LTS (Jammy Jellyfish)",
"family": "debian",
"name": "Ubuntu",
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "jammy",
"type": "linux"
},
"containerized": false,
"ip": [
"172.30.189.193",
"fe80::215:5dff:fecc:d3ab"
],
"mac": [
"00-15-5D-CC-D3-AB",
"5A-2B-CE-65-CF-33",
"62-74-42-62-2C-60"
],
"name": "gab",
"hostname": "gab",
"architecture": "x86_64"
},
"cloud.account.id": "011702-58A742-BEB4E7",
"event": {
"duration": 3895440085,
"dataset": "gcp.billing",
"module": "gcp"
},
"cloud.provider": "gcp",
"ecs": {
"version": "8.0.0"
},
"agent": {
"id": "afdc4ab4-827a-4058-97e7-267618840f1d",
"name": "gab",
"type": "metricbeat",
"version": "8.12.0",
"ephemeral_id": "a071aaea-4ff5-4ec7-b7b0-672f358d2438"
}
}
} Detailed document example:
{
"_index": ".ds-metricbeat-8.12.0-2023.10.19-000001",
"_id": "3h2RSYsBN2wa5YEjUBFz",
"_score": 1,
"_source": {
"@timestamp": "2023-10-19T20:12:17.902Z",
"event": {
"dataset": "gcp.billing",
"module": "gcp",
"duration": 3092063008
},
"metricset": {
"period": 60000,
"name": "billing"
},
"service": {
"type": "gcp"
},
"host": {
"ip": [
"172.30.189.193",
"fe80::215:5dff:fecc:d3ab"
],
"mac": [
"00-15-5D-CC-D3-AB",
"5A-2B-CE-65-CF-33",
"62-74-42-62-2C-60"
],
"name": "gab",
"hostname": "gab",
"architecture": "x86_64",
"os": {
"name": "Ubuntu",
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "jammy",
"type": "linux",
"platform": "ubuntu",
"version": "22.04.2 LTS (Jammy Jellyfish)",
"family": "debian"
},
"containerized": false
},
"gcp": {
"billing": {
"effective_price": 0.00292353,
"tags": {
"compute.googleapis.com/machine_spec": "e2-medium",
"compute.googleapis.com/memory": "4096",
"compute.googleapis.com/cores": "2"
},
"billing_account_id": "011702-58A742-BEB4E7",
"cost_type": "regular",
"sku_id": "F449-33EC-A5EF",
"sku_description": "E2 Instance Ram running in Americas",
"service_description": "Compute Engine",
"invoice_month": "202310",
"project_name": "My Project 70130",
"service_id": "6F81-5844-456A",
"project_id": "robust-catalyst-399814",
"total": 1.700642
}
},
"cloud.provider": "gcp",
"cloud.project.id": "robust-catalyst-399814",
"cloud.account.id": "011702-58A742-BEB4E7",
"agent": {
"ephemeral_id": "dac27eeb-f92c-43be-a0e6-ad3863bc5a27",
"id": "afdc4ab4-827a-4058-97e7-267618840f1d",
"name": "gab",
"type": "metricbeat",
"version": "8.12.0"
},
"ecs": {
"version": "8.0.0"
},
"cloud.project.name": "My Project 70130"
}
} |
Implemented a check to differentiate between standard and detailed cost usage tables Refactoring the 'row' structure used for storing BigQuery results, optimizing data handling Added support for tags Introduced new cost usage data fields
54e857e
to
61e984a
Compare
SQL query is not appropriate for billing data
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change LGTM, but we also need to update the docs.
We need to explain the "standard" vs. "detailed" tables and how to set up the metricset to get one or the other.
We should highlight which extra fields you get from the detailed table.
We should show a sample document and configuration for both standard and detailed cases at https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-gcp-billing.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should update the docs before merging the PR.
The original implementation grouped the data by few fields, resulting in a single document every 24 hours. The updated implementation allows for a more granular view of the data, with each document containing detailed information about each service, SKU etc. In total there are 70 rows, but I only copied the ones that have Updated implementation
Current implementation
|
40f1842
to
eb995a7
Compare
@gpop63 : This is because we are now also grouping by sku.id, service.id etc fields in the query? |
@ishleenk17 Yes, adding more fields to the |
Thanks, I hope this is inline with the grouping activity going on for the TSDB front. |
Shall we create a separate ticket for changes to the dashboards?
Also, some revamping would be needed to the existing dashboard. And add the other visualisations to the dashboard as part of another PR. @zmoog : Thoughts ? |
Yeah, I would make sure the existing dashboard continues to work as expected. We can leverage the new fields in a different PR. |
This pull request is now in conflicts. Could you fix it? 🙏
|
@gpop63 : Lets create a separate ticket for creating updated dashbaords with the new change. |
include tags in the event signature
@gpop63 : Changing the nested to array will bring a change in the geenrated event as well. Lets update that as well. |
@gpop63, why are we changing the tags structure and the aggregation type for the |
x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/057de170-e88d-11ea-bf8c-d13ebf358a78.json
Outdated
Show resolved
Hide resolved
@zmoog If tags field is an object, adding an input control group in the dashboard for filtering based on tags keys and/or values is not possible due to the dynamic mapping. The field definition below enables filtering by tags keys and/or values because there is a mapping which allows us to choose from either - name: tags
type: nested
fields:
- name: key
type: keyword
- name: value
type: keyword |
Thanks for the clarification. |
@gpop63, is this PR going to impact elastic/integrations#5665 or make it obsolete? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets make the PR slight more descriptive regarding the changes in the PR.
Otherwise, looks good!
@zmoog Yeah, kind of. This PR probably made that one obsolete. Also the aggregation change to |
/test |
* GCP Billing Enhancement Implemented a check to differentiate between standard and detailed cost usage tables Refactoring the 'row' structure used for storing BigQuery results, optimizing data handling Added support for tags Introduced new cost usage data fields * add tests fix older tests * add changelog entry * update docs and fields * add sku and service ids in hash * improve docs * update fields types * use array instead of map for tags include tags in the event signature * update sample doc * use sum instead of max for dashboards * fix label minor changes
Overview
Added functionality to check if a billing data table is detailed, using the prefix
gcp_billing_export_resource_v1_
for identification.Added a
row
struct in the code to manage billing data fields, replacing the previous method that accessed fields by indexes. This change makes the data handling clearer and reduces the likelihood of errors due to misindexing.Previously,
max
aggregation displayed daily costs on the billing dashboard for one document per day. With multiple documents now generated daily,sum
aggregation is needed to accurately add the total cost from all documents.Table formats:
gcp_billing_export_v1_<BILLING_ACCOUNT_ID>
gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>
Common fields: Additional fields applicable to both standard and detailed cost data:
sku.id
sku.description
service.id
service.description
tags
Detailed fields:
price.effective_price
Documentation:
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs