Skip to content

Commit 6dbfcc5

Browse files
kgellerefd6
kgeller
and
efd6
authored
[gitlab] Initial release for Gitlab integration (#9747)
* adding integration folder from integration_builder output * filling in actual logo * elastic-package check formatting tweaks * updating integration categories * pipeline tweaks, updated some fields to accommodate * test run outputs * pipeline adjustments and generated outputs * codeowners * filled in and built readme * filled in changelog pr * fixing description, tweaking conditional to be more generic * rebuilding the readme * added in api datastream from integration_builder run * formatting updates from elastic-package check * update to accommodate addition of api datastream * updates to pipelines, fieldname additions, more production samples, test outputs * build, check and format * adding empty line at the end of sample logs * adding api dashboard * adding production dashboard * minor fixes for production ds found when trying additional samples * syncing api to production * results from build, format, check * adding the default options for the filepaths, and some docs to explain * Update packages/gitlab_ce/data_stream/api/agent/stream/filestream.yml.hbs Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab_ce/data_stream/production/agent/stream/filestream.yml.hbs Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab_ce/data_stream/api/elasticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * pipeline updates for converting and formatting; fields updates to be float instead of long for *_duration_s * formatting updates * rebuilt * fixing up convert processors * rebuilt * adding additional system tests for instance created via docker * updating name and fixing formatting * tests actually running now (still failing) * drop processor for info log line * fixed system tests * renaming from gitlab_ce to gitlab; tweaking the readme to better clarify support * missed the rest of the rename * rename updates * updating logo * added back canned system tests * switching around test config * trying adding chmod to GITLAB_PRE_RECONFIGURE_SCRIPT * real path * trying gid * trying group * removing 'real' system tests * removing variants * formatting * syncing manifest title/description to be like github * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/_dev/build/docs/README.md Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Update packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * fixing up the production pipeline like api, rebuilding docs * rebuilt and cleared out the fields we don't need * updated viewBox * Gitlab to GitLab * Gitlab to GitLab --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
1 parent 5bedcd3 commit 6dbfcc5

38 files changed

+9043
-0
lines changed

‎.github/CODEOWNERS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,7 @@
192192
/packages/gcp_metrics @elastic/obs-ds-hosted-services
193193
/packages/gcp_pubsub @elastic/security-service-integrations
194194
/packages/github @elastic/security-service-integrations
195+
/packages/gitlab @elastic/security-service-integrations
195196
/packages/golang @elastic/obs-infraobs-integrations
196197
/packages/google_cloud_storage @elastic/security-service-integrations
197198
/packages/google_scc @elastic/security-service-integrations

‎packages/gitlab/_dev/build/build.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dependencies:
2+
ecs:
3+
reference: "git@v8.11.0"

‎packages/gitlab/_dev/build/docs/README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# GitLab Integration
2+
3+
This integration is for ingesting logs from [GitLab](https://about.gitlab.com/).
4+
5+
- `api`: Collect logs for HTTP requests made to the GitLab API
6+
7+
- `production`: Collect logs for Rails controller requests received from GitLab.
8+
9+
See the GitLab [Log system docs](https://docs.gitlab.com/ee/administration/logs/) for more information.
10+
11+
## Compatibility
12+
13+
The GitLab module has been developed with and tested against the [community edition](https://gitlab.com/rluna-gitlab/gitlab-ce) version 16.8.5-ce.0.
14+
15+
## Setup
16+
17+
Refer to the GitLab documentation for the specific filepath(s) for your instance type. Both are provided as default in the configuration setup, but only one will be needed for use. See [API](https://docs.gitlab.com/ee/administration/logs/#api_jsonlog) and [Production](https://docs.gitlab.com/ee/administration/logs/#production_jsonlog) for details.
18+
19+
## Logs
20+
21+
### api
22+
23+
Collect logs for HTTP requests made to the GitLab API. Check out the [GitLab API log docs](https://docs.gitlab.com/ee/administration/logs/#api_jsonlog) for more information.
24+
25+
{{fields "api"}}
26+
27+
{{event "api"}}
28+
29+
### production
30+
31+
Collect logs for Rails controller requests received from GitLab. Check out the [GitLab production log docs](https://docs.gitlab.com/ee/administration/logs/#production_jsonlog) for more information.
32+
33+
{{fields "production"}}
34+
35+
{{event "production"}}

‎packages/gitlab/_dev/deploy/docker/docker-compose.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
version: '3.0'
2+
services:
3+
gitlab-filestream:
4+
image: alpine
5+
volumes:
6+
- ./sample_logs:/sample_logs:ro
7+
- ${SERVICE_LOGS_DIR}:/var/log
8+
command: /bin/sh -c "cp /sample_logs/* /var/log/"

‎packages/gitlab/_dev/deploy/docker/sample_logs/test-gitlab-api.log

Lines changed: 10 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/gitlab/_dev/deploy/docker/sample_logs/test-gitlab-production.log

Lines changed: 13 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/gitlab/changelog.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# newer versions go on top
2+
- version: 0.1.0
3+
changes:
4+
- description: Initial Version
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/9747

‎packages/gitlab/data_stream/api/_dev/test/pipeline/test-common-config.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
fields:
2+
tags:
3+
- preserve_original_event

‎packages/gitlab/data_stream/api/_dev/test/pipeline/test-gitlab-api.log

Lines changed: 10 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/gitlab/data_stream/api/_dev/test/pipeline/test-gitlab-api.log-expected.json

Lines changed: 1368 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/gitlab/data_stream/api/_dev/test/system/test-filestream-config.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
service: gitlab-filestream
2+
input: filestream
3+
data_stream:
4+
vars:
5+
preserve_original_event: true
6+
paths:
7+
- '{{SERVICE_LOGS_DIR}}/test-gitlab-api.log'
8+
numeric_keyword_fields:
9+
- log.file.device_id
10+
- log.file.inode
11+
- log.file.idxhi
12+
- log.file.idxlo
13+
- log.file.vol
14+
assert:
15+
hit_count: 10

‎packages/gitlab/data_stream/api/agent/stream/filestream.yml.hbs

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
paths:
2+
{{#each paths as |path|}}
3+
- {{path}}
4+
{{/each}}
5+
{{#if exclude_files}}
6+
prospector.scanner.exclude_files:
7+
{{#each exclude_files as |pattern|}}
8+
- {{pattern}}
9+
{{/each}}
10+
{{/if}}
11+
{{#if custom}}
12+
{{custom}}
13+
{{/if}}
14+
tags:
15+
{{#if preserve_original_event}}
16+
- preserve_original_event
17+
{{/if}}
18+
{{#each tags as |tag|}}
19+
- {{tag}}
20+
{{/each}}
21+
{{#contains "forwarded" tags}}
22+
publisher_pipeline.disable_host: true
23+
{{/contains}}
24+
{{#if processors}}
25+
processors:
26+
{{processors}}
27+
{{/if}}

‎packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
---
2+
description: Pipeline to process gitlab api logs
3+
processors:
4+
- set:
5+
field: ecs.version
6+
tag: set_ecs_version
7+
value: 8.11.0
8+
- rename:
9+
field: message
10+
target_field: event.original
11+
tag: rename_message
12+
ignore_missing: true
13+
if: ctx.event?.original == null
14+
- remove:
15+
field: message
16+
ignore_missing: true
17+
tag: remove_message
18+
if: ctx.event?.original != null
19+
- drop:
20+
if: ctx.event.original.startsWith('#')
21+
description: Drop if logline contains header(s), which startswith `#`.
22+
- json:
23+
field: event.original
24+
tag: json_original
25+
target_field: gitlab.api
26+
- date:
27+
field: gitlab.api.time
28+
target_field: '@timestamp'
29+
formats:
30+
- ISO8601
31+
ignore_failure: true
32+
- rename:
33+
field: gitlab.api.severity
34+
target_field: log.level
35+
ignore_missing: true
36+
- script:
37+
lang: painless
38+
source: |-
39+
if (ctx.gitlab?.api?.duration_s != null) {
40+
ctx.event.duration = ctx.gitlab.api.duration_s * 1000000;
41+
}
42+
ignore_failure: true
43+
- rename:
44+
field: gitlab.api.status
45+
target_field: http.response.status_code
46+
ignore_missing: true
47+
- rename:
48+
field: gitlab.api.method
49+
target_field: http.request.method
50+
ignore_missing: true
51+
- rename:
52+
field: gitlab.api.path
53+
target_field: url.path
54+
ignore_missing: true
55+
- rename:
56+
field: gitlab.api.host
57+
target_field: url.domain
58+
ignore_missing: true
59+
- dot_expander:
60+
path: gitlab.api
61+
field: meta.caller_id
62+
- dot_expander:
63+
path: gitlab.api
64+
field: meta.client_id
65+
- dot_expander:
66+
path: gitlab.api
67+
field: meta.feature_category
68+
- dot_expander:
69+
path: gitlab.api
70+
field: meta.remote_ip
71+
- convert:
72+
field: gitlab.api.meta.remote_ip
73+
type: ip
74+
ignore_missing: true
75+
on_failure:
76+
- remove:
77+
field: gitlab.api.meta.remote_ip
78+
ignore_missing: true
79+
- append:
80+
field: error.message
81+
value: '{{{_ingest.on_failure_message}}}'
82+
- dot_expander:
83+
path: gitlab.api
84+
field: meta.user
85+
- dot_expander:
86+
path: gitlab.api
87+
field: meta.user_id
88+
- rename:
89+
field: gitlab.api.meta.caller_id
90+
target_field: event.provider
91+
ignore_missing: true
92+
- split:
93+
field: gitlab.api.remote_ip
94+
if: ctx.gitlab?.api?.remote_ip != null && ctx.gitlab?.api?.remote_ip != ''
95+
tag: split_ips
96+
target_field: source.ip
97+
separator: ', ?'
98+
- foreach:
99+
field: source.ip
100+
if: ctx.source?.ip instanceof List
101+
ignore_failure: true
102+
processor:
103+
convert:
104+
field: _ingest._value.name
105+
type: ip
106+
target_field: _ingest._value.name_ips
107+
ignore_missing: true
108+
on_failure:
109+
- remove:
110+
field: _ingest._value.ip
111+
ignore_missing: true
112+
- remove:
113+
field: gitlab.api.remote_ip
114+
ignore_missing: true
115+
- rename:
116+
field: gitlab.api.ua
117+
target_field: user_agent.original
118+
ignore_missing: true
119+
- rename:
120+
field: gitlab.api.pid
121+
target_field: process.pid
122+
ignore_missing: true
123+
- rename:
124+
field: gitlab.api.user_id
125+
target_field: user.id
126+
ignore_missing: true
127+
- convert:
128+
field: user.id
129+
type: string
130+
ignore_missing: true
131+
- rename:
132+
field: gitlab.api.username
133+
target_field: user.name
134+
ignore_missing: true
135+
- geoip:
136+
field: client.address
137+
tag: geoip_client_address
138+
target_field: client.geo
139+
ignore_missing: true
140+
- geoip:
141+
ignore_missing: true
142+
database_file: GeoLite2-ASN.mmdb
143+
field: client.address
144+
tag: geoip_client_asn
145+
target_field: client.as
146+
properties:
147+
- asn
148+
- organization_name
149+
- rename:
150+
field: client.as.asn
151+
tag: rename_client_as_asn
152+
target_field: client.as.number
153+
ignore_missing: true
154+
- rename:
155+
field: client.as.organization_name
156+
tag: rename_client_as_organization_name
157+
target_field: client.as.organization.name
158+
ignore_missing: true
159+
- append:
160+
field: event.category
161+
value:
162+
- api
163+
allow_duplicates: false
164+
- append:
165+
field: event.type
166+
value:
167+
- info
168+
allow_duplicates: false
169+
- append:
170+
field: event.category
171+
value:
172+
- database
173+
allow_duplicates: false
174+
if: ctx.gitlab?.api?.db_count != null && ctx.gitlab?.api?.db_count > 0
175+
- append:
176+
field: event.type
177+
value:
178+
- error
179+
allow_duplicates: false
180+
if: ctx.http?.response?.status_code != null && (ctx.http.response.status_code < 200 || ctx.http.response.status_code >= 400)
181+
- append:
182+
field: event.kind
183+
value:
184+
- pipeline_error
185+
allow_duplicates: false
186+
if: ctx.error?.message != null
187+
- remove:
188+
field: event.original
189+
tag: remove_original_event
190+
if: ctx?.tags == null || !(ctx.tags.contains("preserve_original_event"))
191+
ignore_failure: true
192+
ignore_missing: true
193+
- script:
194+
description: Drops null/empty values recursively.
195+
lang: painless
196+
source: |
197+
boolean dropEmptyFields(Object object) {
198+
if (object == null || object == "") {
199+
return true;
200+
} else if (object instanceof Map) {
201+
((Map) object).values().removeIf(value -> dropEmptyFields(value));
202+
return (((Map) object).size() == 0);
203+
} else if (object instanceof List) {
204+
((List) object).removeIf(value -> dropEmptyFields(value));
205+
return (((List) object).length == 0);
206+
}
207+
return false;
208+
}
209+
dropEmptyFields(ctx);
210+
on_failure:
211+
- append:
212+
field: error.message
213+
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}}
214+
in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
215+
- set:
216+
field: event.kind
217+
value: pipeline_error

‎packages/gitlab/data_stream/api/fields/agent.yml

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
- name: cloud
2+
title: Cloud
3+
group: 2
4+
description: Fields related to the cloud or infrastructure the events are coming from.
5+
footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.'
6+
type: group
7+
fields:
8+
- name: image.id
9+
type: keyword
10+
description: Image ID for the cloud instance.
11+
- name: container
12+
title: Container
13+
group: 2
14+
description: 'Container fields are used for meta information about the specific container that is the source of information. These fields help correlate data based containers from any runtime.'
15+
type: group
16+
fields:
17+
- name: labels
18+
level: extended
19+
type: object
20+
object_type: keyword
21+
description: Image labels.
22+
- name: host
23+
title: Host
24+
group: 2
25+
description: 'A host is defined as a general computing instance. ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
26+
type: group
27+
fields:
28+
- name: containerized
29+
type: boolean
30+
description: >
31+
If the host is a container.
32+
33+
- name: os.build
34+
type: keyword
35+
example: "18D109"
36+
description: >
37+
OS build information.
38+
39+
- name: os.codename
40+
type: keyword
41+
example: "stretch"
42+
description: >
43+
OS codename, if any.
44+

‎packages/gitlab/data_stream/api/fields/base-fields.yml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
- name: data_stream.type
2+
type: constant_keyword
3+
description: Data stream type.
4+
- name: data_stream.dataset
5+
type: constant_keyword
6+
description: Data stream dataset name.
7+
- name: data_stream.namespace
8+
type: constant_keyword
9+
description: Data stream namespace.
10+
- name: event.module
11+
type: constant_keyword
12+
description: Event module
13+
value: gitlab
14+
- name: event.dataset
15+
type: constant_keyword
16+
description: Event dataset
17+
value: gitlab.api
18+
- name: "@timestamp"
19+
type: date
20+
description: Event timestamp.

0 commit comments

Comments
 (0)