Releases: elastic/ecs
ECS 1.6.0
This release adds the x509.*
field set to capture common core fields for x509 certificates. Other notable schema changes include the introduction of event.reason
, adding span.id
to the transaction.*
field set, and new related.*
fields. Please see the full schema change details below.
Before this release, there was no way to reuse field sets as different names inside themselves. Now nesting fields within themselves, such as process
=> process.parent
, and defining nested sets using a different name are both available.
Did you know you can use the Python scripts in the ECS repository to generate Elasticsearch templates containing the only ECS fields you need + your custom fields? A lot of the changes in the "tooling and artifact" changelog below are about how we improved this experience. However you can jump directly to the new usage documentation to learn how to do this.
Finally in previous releases, reusable fields not expected at the root of documents were accidentally defined at the root in some generated artifacts. This incorrect behavior is fixed in this release.
Schema Changes
Bugfixes
- Field
registry.data.strings
should have been marked as an array field. #790
Added
- Added
x509.*
field set. #762 - Add architecture and imphash for PE field set. #763
- Added
agent.build.*
for extended agent version information. #764 - Added
log.file.path
to capture the log file an event came from. #802 - Added more account and project cloud metadata. #816
- Added missing field reuse of
pe
atprocess.parent.pe
#868 - Added
span.id
to the tracing fieldset, for additional log correlation #882 - Added
event.reason
for the reason why an event's outcome or action was taken. #907 - Added
related.hosts
to capture all hostnames and host identifiers on an event. #913 - Added
user.roles
to capture a list of role names that apply to the user. #917
Improvements
- Removed misleading pluralization in the description of
user.id
, it should
contain one ID, not many. #801 - Clarified misleading wording about multiple IPs in src/dst or cli/srv. #804
- Improved verbiage about the MITRE ATT&CK® framework. #866
- Removed the default
object_type=keyword
that was being applied toobject
fields.
This attribute is Beats-specific. It's still supported, but needs to be set explicitly
on a case by case basis now. This default being removed affectsdns.answers
,
log.syslog
,network.inner
,observer.egress
, andobserver.ingress
. #871 - Improved attribute
dashed_name
ingenerated/ecs/*.yml
to also
replace@
with-
. #871 - Updated several URLs in the documentation with "example.com" domain. #910
Deprecated
- Deprecate guidance to lowercase
http.request.method
#840
Tooling and Artifact Changes
Breaking changes
- Removed field definitions at the root of documents for fieldsets that
hadreusable.top_level:false
. This PR affectsecs_flat.yml
, the csv file
and the sample Elasticsearch templates. #495, #813 - Removed the
order
attribute from theecs_nested.yml
andecs_flat.yml
files. #811 - In
ecs_nested.yml
, the array of strings that used to be inreusable.expected
has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864 - The subset format now requires
name
andfields
keys at the top level. #873
Bugfixes
- Subsets are created after duplicating reusable fields now so subsets can
be applied to each reused instance independently. #753 - Quoted the example for
labels
to avoid YAML interpreting it, and having
slightly different results in different situations. #782 - Fix incorrect listing of where field sets are nested in asciidoc,
when they are nested deep. #784 - Allow beats output to be generated when using
--include
or--subset
flags. #814 - Field parameter
index
is now correctly populated in the Beats field definition file. #824
Improvements
- Add support for reusing official fieldsets in custom schemas. #751
- Add full path names to reused fieldsets in
nestings
array inecs_nested.yml
. #803 - Allow shorthand notation for including all subfields in subsets. #805
- Add support for Elasticsearch
enabled
field parameter. #824 - Add
ref
option to generator allowing schemas to be built for a specific ECS version. #851 - Add
template-settings
andmapping-settings
options to allow override of defaults in generated ES templates. #856 - When overriding ECS field sets via the
--include
flag, it's no longer necessary
to duplicate the field set's mandatory attributes. The customizations are merged
before validation. #864 - Add ability to nest field sets as another name. #864
- Add ability to nest field sets within themselves (e.g.
process
=>process.parent
). #864 - New attribute
reused_here
is added inecs_nested.yml
. It obsoletes the
previous attributenestings
, and is able to fully capture details of other
field sets reused under this one. #864 - When chained reuses are needed (e.g.
group
=>user
, thenuser
=> many places),
it's now necessary to force the order with new attributereusable.order
. This
attribute is otherwise optional. It's currently only needed forgroup
. #864 - There's a new representation of ECS at
generated/ecs/ecs.yml
, which is a deeply nested
representation of the fields. This file is not in git, as it's only meant for
developers working on the ECS tools. #864 - Jinja2 templates now define the doc structure for the AsciiDoc generator. #865
- Intermediate
ecs_flat.yml
andecs_nested.yml
files are now generated for each individual subset,
in addition to the intermediate files generated for the combined subset. #873
Deprecated
- In
ecs_nested.yml
, we're deprecating the attributenestings
. It will be
removed in a future release. The deprecatednestings
attribute was an array of
flat field names describing where fields are nested within the field set.
This is replaced with the attributereused_here
, which is an array of objects.
The new format still lists where the fields are nested via the same flat field name,
but also specifies additional information about each field reuse. #864
ECS 1.5.0
In this release, we continue fleshing out categorization by introducing the "network" and "iam" categories, with related event types.
We're adding new field sets: "dll", "pe", "code_signature", "interface" & "vlan". We're also adding a few fields here and there (check out the details below).
Implementers consuming ECS artifacts like generated/ecs/*.yml
programmatically will be happy to know that we now clearly identify which fields are expected to contain an array of values. Shout-out to contributors on the ecs-logging libraries for raising this 👋🏼.
Finally, starting with ECS 1.5.0, the project is using Python 3.7.
Schema Changes
Added
- Added
dll.*
fields #679 - Added
related.hash
to keep track of all hashes seen on an event. #711 - Added fieldset for PE metadata. #731
- Added
code_signature
fieldset. #733 - Added missing
hash
fields atprocess.parent.hash.*
. #739 - Added globally unique identifier
entity_id
toprocess
andprocess.parent
. #747 - Added interface, vlan, observer zone fields #752
- Added
rule.author
,rule.license
fields #754 - Added iam value for
event.category
and three related values forevent.type
. #756 - Added fields
event.reference
andevent.url
to hold link to additional event info/actions. #757 - Added
file.mime_type
to include MIME type information on file structures #760 - Added
event.category
value of network and associatedevent.type
values. #761
Improvements
- Temporary workaround for Beats templates'
default_field
growing too big. #687 - Identify which fields should contain arrays of values, rather than scalar values. #727, #661
- Clarified examples and definitions regarding vulnerabilities. #758
- Updated definition of
event.outcome
based on community feedback. #759
Tooling and Artifact Changes
Improvements
- ECS scripts now use Python 3.6+. #674
- schema_reader.py now reliably supports chaining reusable fieldsets together. #722
- Allow the artifact generator to consider and output only a subset of fields. #737
- Add support for reusing fields in places other than the top level of the destination fieldset. #739
- Add support for specifying the directory to write the generated files. #748
ECS 1.4.0
This release introduces two much-awaited changes.
The text
analyzer has been added to many existing fields. This enables full text search queries on fields that contain a lot of text, or semi-structured data (such as file paths and urls). Look at #575 and #680 to learn more. As an example, the field user_agent.original
can now service full text search queries at user_agent.original.text
.
We're also introducing the first set of allowed values for the 4 previously reserved fields (event.kind
, event.category
, event.type
and event.outcome
). We're calling them the "categorization fields". More allowed values will be released over time. You can preview future values, and provide feedback in this public document: https://ela.st/ecs-categories-draft. Learn more in the new "ECS Categorization Fields" section of the documentation.
Schema Changes
Added
- Added default
text
analyzer as a multi-field touser_agent.original
. #575 - Added
file.attributes
. #611 - Added
file.drive_letter
. #620 - Added
rule
fields. #665 - Added default
text
analyzer as a multi-field to around 25 more fields. #680 - Added
registry.*
fieldset for the Windows registry. #673 - Publish initial list of allowed values for the categorization fields (previously reserved)
event.kind
,event.category
,event.type
andevent.outcome
. #684, #691, #692 - Added
related.user
#694
Tooling and Artifact Changes
Bugfixes
- Fix support for multi-fields. #575
ECS 1.3.1
ECS 1.3.0
Schema Changes
Added
- Added
vulnerability.*
fields to represent vulnerability information. #581 - Added
event.ingested
as the ingest timestamp. #582 - Added
package.reference
. #585 - Added
package.build_version
. #586 - Added
package.type
. #587 - Added
host.domain
field. #591 - Added
process.command_line
. #599 - Added
process.exit_code
. #600 - Added fields in
tls.*
to support analysis of TLS protocol events. #606 - Added
process.parent.*
. #612 - Added
process.args_count
. #615
Tooling and Artifact Changes
Breaking changes
- Changed the order and column names in the csv. #621
- Removed the file
schema.json
and the code generating it. #627 - Removed the legacy Elasticsearch template. #629
- Note: The good Elasticsearch templates are available in directory
generated/elasticsearch
, this PR only removes an obsolete file.
- Note: The good Elasticsearch templates are available in directory
Added
- Added the "Indexed", "Field_Set" and "Description" columns to the csv. #621
ECS 1.2.0
Added
- Added
threat.*
fields to apply a taxonomy to events and alerts. #505 - Added fields in
log.*
to allow for full Syslog mapping. #525 - Added
package.*
to installed software packages. #532 - Added
registered_domain
tourl
,source
,destination
,client
, andserver
. #533 - Added
top_level_domain
field tourl
,dns.question
,source
,destination
,client
, andserver
. #542, #572 - Added
group.domain
field. #547 - Added
url.extension
. #551, #573 - Added
observer.name
andobserver.product
. #557, #571 - Added
dns.question.subdomain
field. #561, #574 - Added
error.stack_trace
field. #562 - Added
log.origin.file.name
,log.origin.function
andlog.origin.file.line
fields. #563, #568 - Added
service.node.name
to allow distinction between different nodes of the same service running on the same host. #565 - Added
error.type
field. #566
ECS 1.1.0
Added
- Added
as
fields for Autonomous System information (i.e. ASN). #341 - Added field formats to all
.bytes
fields andevent.duration
. #385, #425 - Added
hash.*
field set. #426 - Added
dns.*
field set, to describe DNS traffic. #438 - Added
event.code
,event.sequence
andevent.provider
. #439 - Added
file.name
andfile.directory
. #441 - Added
file.created
, andfile.accessed
. #445 - Added
process.uptime
andhost.uptime
fields. #477 - Added
domain
field to user. #486 - Added
.nat.ip
and.nat.port
tosource
,destination
,client
andserver
. #491 - Added
process.thread.name
field. #517 - Added
trace.id
andtransaction.id
fields for tracing across different services. #519 - Added
log.logger
field. #521
Improvements
ECS 1.0.1
Added
- Add generated source code for Go. #249
- Translate the documentation from README.md, to the main website. #266, #334, #400, #430, #435
- New generator that supports reusable fields, for files based on ECS.
It generates schema.csv, Elasticsearch 6 and 7 templates, and field documentation
for the main website. #336 - Generator for the asciidoc rendering of field definitions. #347
- Generator for the Beats fields.ecs.yml file. #379
- Remove many legacy generated files. #399
- Specify static output format for event.duration. #425
- Format port numbers and numeric IDs as strings. #454
- Add example for
process.pid
andprocess.ppid
. #464, #470
ECS 1.0.0
ECS is turning 1.0.0!
We've clarified a few field descriptions and examples, based on your feedback and questions.
Breaking changes
- Remove the
user.group
keyword
field, introduced in #204. Instead,
thegroup
field set can be nested atuser.group
. #308
Bugfixes
- Field set name "group" was being used as a leaf field at
user.group
, instead
of being a nesting of the field set. This goes against a driving principle of ECS,
and has been corrected. #308 - Replaced incorrect examples in
cloud.provider
. #330, #348 - Changed the
url.port
type tolong
. #339
Added
- Added pointer in description of
http
field set tourl
field set. #330 - Added an optional short field description. #330
Improvements
- Clarified the definition of the host fields #325
- Clarified the difference between
@timestamp
andevent.created
. #329 - Make phrasing of lowercasing directive more relevant, no matter where it's shown. #332
- Specify the
object_type
for fieldlabels
. #331 - Loosen up definition of
geo
field set. Not necessarily geo-ip based, sincegeo.name
. #333 - Clarified guidelines on ID fields. #349
ECS 1.0.0-beta2
This is the second 1.0.0 pre-release of ECS. From 1.0.0-beta2 to 1.0.0 GA, no further breaking changes and no additions or new fields are planned. All new contributions must go into the master branch.
Breaking changes
- Changed
device.*
fields toobserver.*
fields to eliminate user confusion. #238 - Rename
network.total.bytes
tonetwork.bytes
andnetwork.total.packets
tonetwork.packets
. #179 - Remove
network.inbound.bytes
,network.inbound.packets
,
network.outbound.bytes
andnetwork.outbound.packets
. #179 - Changed the
event.type
definition to be only reserved. #242
Bugfixes
- Fix obvious mistake in the definition of "source", where it said "destination"
instead of "source". #211
Added
- Add
host.name
field and clarify usage ofhost.hostname
. #187 - Add
event.start
andevent.end
date fields. #185 - Add
process.thread.id
field. #200 - Add
host.name
field and clarify usage ofhost.hostname
. - Add
event.start
andevent.end
date fields. - Create new
related
field set withrelated.ip
. #206 - Add
user.group
field. #204 - Create new
group
field set withgroup.id
andgroup.name
. #203 - Add
url.full
field. #207 - Add
process.executable
field. #209 - Add
process.working_directory
andprocess.start
. #215 - Reintroduce
http
. #237 - Add
user.full_name
field. #201 - Add
network.community_id
field. #208 - Add fields
geo.country_name
andgeo.region_iso_code
. #214 - Add
event.kind
andevent.outcome
. #242 - Add
client
andserver
objects and fields. #236 - Reintroduce a streamlined
user_agent
field set. #240, #262 - Add
geo.name
for ad hoc location names. #248 - Add
event.timezone
to allow for proper interpretation of incomplete timestamps. #258 - Add fields
source.address
,destination.address
,client.address
, and
server.address
. #247 - Add
os.full
to capture full OS name, including version. #259
Improvements
- Improved the definition of the file fields #196
- Improved the definition of the agent fields #192
- Improve definition of events, logs, and metrics in event section #194
- Improved the definition of network fields in intro section #197
- Improved the definition of host fields #195
- Improved the definitions for
event.category
andevent.action
. #242 - Clarify the semantics of
network.direction
. #212 - Add
source.bytes
,source.packets
,destination.bytes
anddestination.packets
. #179 - Add a readme section to declare some top level field sets are reserved for
future use. #257 - Clarify that
network.transport
,network.type
,network.application
,
andnetwork.protocol
must be lowercase. #251 - Clarify that
http.request.method
must be lowercase. #251 - Clarify that source/destination should be filled, even if client/server is
being used. #265