Releases: elastic/ecs
ECS 8.0.1
ECS 8.0.0
What's new in ECS 8.0
We're pleased to announce ECS 8.0.
Thank you to all the ECS contributors who help support the broader Elastic community.
Versioning: 1.x -> 8.0
ECS versioning now aligns with the Elastic platform beginning with 8.0.
ECS didn't follow the same release cadence as the Elastic platform when first introduced. Over time this approach added complexity for our users. For example, users might find themselves asking, "which Elastic version maps to ECS 1.6.0?". By aligning, it's clear what version of ECS maps to which Elastic platform version.
Power in simplicity. 😃
Removed fields
The following fields are removed in ECS 8.0:
Field | Migrate to* | Reference |
---|---|---|
log.original |
event.original |
RFC 0017 |
process.ppid |
process.parent.pid |
RFC 0022 |
host.user.* reuse |
user.* reuses |
user.* field set usage |
*Field aliases can help transition existing searches or visualizations depending on these removed fields.
New field data types
ECS 1.x introduced wildcard
and match_only_text
as beta field types. As of ECS 8.0, these data types are now GA.
The field types selected for ECS provide the best default experience for most users. However, some users may see interoperable data types better fitting for their use cases, and they can read more about options here.
Tooling changes
Elasticsearch generated artifacts
In 1.x, the project maintained sample index templates for two versions of Elasticsearch (6.x, 7.x). In 8.0, ECS now produces two sample template types: composable
and legacy.
In composable,
each ECS field set has a component template. An example component template, template.json,
references each field set component template. These artifacts work with the new index templates introduced in Elasticsearch 7.8.
The legacy
template will continue working with the legacy index template API.
Removed features
- Removed the already deprecated
--oss
flag - Removed Go code generator to simplify the project's tooling and CI/CD pipeline.
Changelog
Schema Changes
Breaking changes
- Remove
host.user.*
field reuse. #1439 - Remove deprecation notice on
http.request.method
. #1443 - Migrate
log.origin.file.line
frominteger
tolong
. #1533 - Remove
log.original
field. #1580 - Remove
process.ppid
field. #1596
Added
Improvements
- Wildcard type field migration GA. #1582
match_only_text
type field migration GA. #1584- Threat indicator fields GA from RFC 0008. #1586
Tooling and Artifact Changes
Breaking Changes
- Removing deprecated --oss from generator #1404
- Removing use-cases directory #1405
- Remove Go code generator. #1567
- Remove template generation for ES6. #1680
- Update folder structure for generated ES artifacts. #1700, #1762
- Updated support for overridable composable settings template. #1737
Improvements
- Align input options for --include and --subset arguments #1519
- Remove remaining Go deps after removing Go code generator. #1585
- Add explicit
default_field: true
for Beats artifacts. #1633 - Reorganize docs directory structure. #1679
- Added support for
analyzer
definitions for text fields. #1737
Bugfixes
- Fixed the
default_field
flag for root fields in Beats generator. #1711
ECS 1.12.2
ECS 1.12.1
ECS 1.12.0
The following RFCs have advanced as a part of this release:
Stage 3 (GA)
- RFC 0018 - extend
threat.*
field set - RFC 0001 - wildcard field migration
- RFC 0023 - migrate
text
tomatch_only_text
type
Stage 2 (beta)
Stage 1 (experimental)
There's also been a couple of new field additions in 1.12: file.fork_name
, service.address
, process.end
, code_signature.digest_algorithm
and code_signature.timestamp
.
Lastly, a couple tooling and documentation improvements. There now exists support for multi-field type fallback to better support ES 6 types as well as the new match_only_text
type. And finally, we updated examples within user
to better clarify things.
Changelog
Schema Changes
Bugfixes
- Updating
hash
order to correct nesting. #1603 - Removing incorrect
hash
reuses. #1604 - Updating
pe
order to correct nesting. #1605 - Removing incorrect
pe
reuses. #1606 - Correcting
enrichments
to anarray
type. #1608
Added
- Added
file.fork_name
field. #1288 - Added
service.address
field. #1537 - Added
service.environment
as a beta field. #1541 - Added
process.end
field. #1544 - Added container metric fields into experimental schema. #1546
- Add
code_signature.digest_algorithm
andcode_signature.timestamp
fields. #1557 - Add
email.*
field set in the experimental fields. #1569
Improvements
- Beta migration on some
keyword
fields towildcard
. #1517 - Promote
threat.software.*
andthreat.group.*
fields to GA. #1540 - Update
user.name
anduser.id
examples for clarity. #1566 - Beta migration of
text
and.text
multi-fields tomatch_only_text
. #1532, #1571
Tooling and Artifact Changes
Added
- Support ES 6.x type fallback for
match_only_text
field types. #1528
Bugfixes
- Prevent failure if no files need to be deleted
find | xargs rm
. #1588
Improvements
- Document field type family interoperability in FAQ. #1591
ECS 1.11.0
The following RFCs have advanced as part of this release:
Stage 3 (GA)
Stage 2 (beta)
- RFC 0008 - Threat indicator fields
- RFC 0015 -
elf
file fields - RFC 0018 - Extend the
threat.*
field set withthreat.software.*
andthreat.group.*
fields - RFC 0021 - Threat enrichment
Stage 1 (experimental)
The event.agent_id_status
field is also new in 1.11 to reflect the status of the agent.id
verification performed by a receiving system or data pipeline.
Lastly, many tooling and documentation improvements, including the --exclude
flag. The --exclude
flag adds the ability to remove individual fields from the schema. More detail is available in the usage doc.
Changelog
Schema Changes
Added
elf.*
field set added as beta. #1410- Remove
beta
fromorchestrator
field set. #1417 - Extend
threat.*
field set beta. #1438 - Added
event.agent_id_status
field. #1454 process.target
andprocess.target.parent
added to experimental schema. #1467- Threat indicator fields progress to beta stage. #1471, #1504
threat.enrichments
beta fields. #1478, #1504
Improvements
- Fix ecs GitHub repo link source branch #1393
- Add --exclude flag to Generator to support field removal testing #1411
- Explicitly include user identifiers in
relater.user
description. #1420 - Improve descriptions for
cloud.region
andcloud.availability
fields. #1452 - Clarify
event.kind
descriptions foralert
andsignal
. #1548
Deprecated
- Note deprecation of the
host.user.*
field reuse. #1422 - Note deprecation of
log.original
superseded byevent.original
#1469
Tooling and Artifact Changes
Bugfixes
- Remove
ignore_above
whenindex: false
anddoc_values: false
. #1483 - Ensure
doc_values
is carried into Beats artifacts. #1488
Added
- Support
match_only_text
data type in Go code generator. #1418 - Support for multi-level, self-nestings. #1459
beta
attribute now supported on categorization allowed values. #1511
Improvements
ECS 1.10.0
A handful of new additions from the ECS RFC process are included in this release:
- The host metrics RFC has advanced to Finished status with host metrics fields becoming GA.
- The orchestrator fieldset RFC has advanced to Stage 3, and the fieldset has been released for beta.
- The
data_stream
fields moved to Stage 2, and are released for beta. - We are extending the existing `threat.* fields, which are released as experimental.
In addition to RFC proposed changes, ECS 1.10.0 also adds some documentation updates, including the ability to add a short_override
to field reuses for a custom description.
Finally, there is now support for flattened and nested types in the Go code generator script.
Changelog
Schema Changes
Added
- Add
data_stream
fieldset. #1307 - Add
orchestrator
fieldset as beta fields. #1326 - Extend
threat.*
experimental fields with proposed changes from RFC 0018. #1344, #1351 - Allow custom descriptions for self-nesting reuses via
short_override
#1366
Improvements
- Updated descriptions to use Elastic Security #1305
- Host metrics fields from RFC 0005 are now GA. #1319
- Adjustments to the field set "usage" docs #1345
- Adjustments to the sidebar naming convention for usage and examples docs #1354
- Update
user.*
field reuse descriptions. #1382
Tooling and Artifact Changes
Bugfixes
- Correcting fieldset name capitalization for generated ES template #1323
Improvements
ECS 1.9.0
Several additions introduced from the ECS RFC process are included in this release:
- The multiple users proposal has advanced to
Finished
status withuser.changes.*
,user.effective.*
, anduser.target.*
field reuses becoming GA. - Host metrics fields are now beta.
- The
threat.indicator
fields,elf.*
fields,pe.*
extensions, anddata_stream.*
fieldset are now in the experimental ECS schema.
A new section has been added to the ECS event categorization documentation. Real-world example events are categorized to demonstrate using the event categorization fields to group and identify similar events from multiple data sources.
In addition to RFC proposed changes, ECS 1.9.0 also adds:
http.request.id
cloud.service.name
hash.ssdeep
code_signature.team_id
andcode_signature.signing_id
- Additional fields to the
geo.*
fieldset:geo.timezone
,geo.postal_code
,geo.continent_code
Finally, *.mac
field descriptions now suggest normalizing MAC address values to the RFC7042 format.
Changelog
Schema Changes
Added
- Added
hash.ssdeep
. #1169 - Added
cloud.service.name
. #1204 - Added
http.request.id
. #1208 data_stream.*
fieldset introduced in experimental schema and artifacts. #1215- Added
geo.timezone
,geo.postal_code
, andgeo.continent_code
. #1229 - Added
beta
host metrics fields. #1248 - Added
code_signature.team_id
,code_signature.signing_id
. #1249 - Extended
pe
fields added to experimental schema. #1256 - Add
elf
fieldset to experimental schema. #1261 - Add
threat.indicator
fields to experimental schema. #1268
Improvements
- Include formatting guidance and examples for MAC address fields. #456
- New section in ECS detailing event categorization fields usage. #1242
user.changes.*
,user.effective.*
, anduser.target.*
field reuses are GA. #1271
Tooling and Artifact Changes
Improvements
ECS 1.8.0
In this release, two ECS RFCs are advancing. The multiple users in an event RFC proposed field reuses now appear in the ECS documentation as beta. The host metrics fields are also advancing and are available in the experimental schema and artifacts.
Accompanying the multiple user
changes, the user.*
fieldset adds ECS' first usage doc. The user usage page contains guidance on categorization, user ids, field reuse, and mapping examples.
The event categorization fields, with the initial set of allowed values, were introduced as beta in ECS 1.4.0. Over the past several ECS released, we've iterated and further fleshed out these fields and values. We're excited to announce that the event categorization fields are now generally available!
In addition to the event categorizations fields becoming GA, two additional event.category
allowed values have also been introduced: registry
and session.
A new field, os.type
, is intended to ease filtering for Windows, Unix, Linux, and macOS events.
Finally, a component template and composable templates (per fieldset) have been added as generated artifacts. The legacy index templates for Elasticsearch 6.x and 7.x are still being maintained. More details covered here.
Changelog
Schema Changes
Bugfixes
- Clean up
event.reference
description. #1181 - Go code generator fails if
scaled_float
type is used. #1250
Added
- Added
event.category
"registry". #1040 - Added
event.category
"session". #1049 - Added usage documentation for
user
fields. #1066 - Added
user
fields atuser.effective.*
,user.target.*
anduser.changes.*
. #1066 - Added
os.type
. #1111
Improvements
- Event categorization fields GA. #1067
- Note
[
and]
bracket characters may enclose a literal IPv6 address when populatingurl.domain
. #1131 - Reinforce the exclusion of the leading dot from
url.extension
. #1151
Deprecated
- Deprecated
host.user.*
fields for removal at the next major. #1066
Tooling and Artifact Changes
Bugfixes
tracing
fields should be at root of Beatsfields.ecs.yml
artifacts. #1164
Added
- Added the
path
key when type isalias
, to support the alias field type. #877 - Added support for
scaled_float
's mandatory parameterscaling_factor
. #1042 - Added ability for --oss flag to fall back
constant_keyword
tokeyword
. #1046 - Added support in the generated Go source go for
wildcard
,version
, andconstant_keyword
data types. #1050 - Added support for marking fields, field sets, or field reuse as beta in the documentation. #1051
- Added support for
constant_keyword
's optional parametervalue
. #1112 - Added component templates for ECS field sets. #1156, #1186, #1191
- Added functionality for merging custom and core multi-fields. #982
Improvements
ECS 1.7.0
Experimental Changes
A few months ago, we introduced the RFC process. This process is meant to fully vet big additions or changes to ECS. A key aspect of this process is that proposals advance in stages. Each stage represents the vetting and maturity of the proposal.
We won’t go over the process in detail here, but one of its key aspects is that accepted “stage 2” proposals appear in “experimental” ECS artifacts. They don’t yet appear officially in ECS documentation. Proposals that reach “stage 3” are the ones that will officially appear in ECS documentation.
ECS 1.7 is the first release that includes RFCs that have reached stage 2 / experimental changes. A new directory has therefore been added, where all the usual generated artifacts are published including the experimental changes. This is at experimental/generated.
This release includes experimental changes from two RFCs reaching stage 2:
- Replace the
keyword
type on many existing ECS fields with the newwildcard
type. - Adding more places where user fields can be nested, in order to capture privilege escalations & demotions as well as IAM. These experimental nestings are
user.effective.*
,user.target.*
, anduser.changes.*
.
“Normal” Changes
Contrary to the new experimental changes described above, the following changes are reflected in the documentation.
Two new fields are introduced: http.[request|response].mime_type
/ and threat.technique.subtechnique
.
Both the network.direction
and event.category
fields add support for additional allowed values.
The ECS generator script adds two new arguments, --oss
and --strict
. See usage for more details and examples.
Lastly, we have changed the index pattern of the sample Elasticsearch template from ecs-*
to try-ecs-*
to avoid conflicting with Logstash' template when run in ECS compatibility mode.
Changelog
Schema Changes
Bugfixes
- The
protocol
allowed value underevent.type
should not have theexpected_event_types
defined. #964 - Clarify the definition of
file.extension
(no dots). #1016
Added
- Added Mime Type fields to HTTP request and response. #944
- Added network directions ingress and egress. #945
- Added
threat.technique.subtechnique
to capture MITRE ATT&CK® subtechniques. #951 - Added
configuration
as an allowedevent.category
. #963 - Added a new directory with experimental artifacts, which includes all changes
from RFCs that have reached stage 2. #993, #1053, #1115, #1117, #1118
Improvements
- Expanded field set definitions for
source.*
anddestination.*
. #967 - Provided better guidance for mapping network events. #969
- Added the field
.subdomain
underclient
,destination
,server
,source
andurl
, to match its presence atdns.question.subdomain
. #981 - Clarified ambiguity in guidance on how to use x509 fields for connections with
only one certificate. #1114
Tooling and Artifact Changes
Breaking changes
- Changed the index pattern of the sample Elasticsearch template from
ecs-*
to
try-ecs-*
to avoid conflicting with Logstash'ecs-logstash-*
. #1048
Bugfixes
- Addressed issue where foreign reuses weren't using the user-supplied
as
value for their destination. #960 - Experimental artifacts failed to install due to
event.original
index setting. #1053
Added
- Introduced
--strict
flag to perform stricter schema validation when running the generator script. #937 - Added check under
--strict
that ensures composite types in example fields are quoted. #966 - Added
ignore_above
andnormalizer
support for keyword multi-fields. #971 - Added
--oss
flag for users who want to generate ECS templates for use on OSS clusters. #991