The n6 project uses a versioning scheme
distinct from Semantic Versioning. Each n6 version's identifier
consists of three integer numbers, separated with .
(e.g.: 4.11.2
).
We can say it is in the <FOREMOST>.<MAJOR>.<MINOR>
format -- where:
-
<MINOR>
is incremented on changes that are backwards compatible from the point of view of users, sysadmins and backend programmers. Note that such changes may still be backwards incompatible regarding any code or feature which is considered non-public or experimental (by convention or because it is explicitly marked as such), any n6 Portal frontend internals (all JS/TS code and all HTML/CSS/etc.) as well as the documentation/experimentation/examples-focused stuff in thedocker/
,docs/
andetc/
directories. -
<MAJOR>
is incremented on more significant changes -- which typically are backwards incompatible from the point of view of users, sysadmins or backend programmers. -
<FOREMOST>
is incremented very rarely, only for big milestone releases.
Some features of this document's layout were inspired by Keep a Changelog.
4.12.0 (2024-12-23)
-
[setup, lib, etc/docker] Debian 12 (bookworm) and CPython 3.11 are now the officially recommended operating system and Python implementation.
-
[data sources] New data sources:
turris-cz.greylist-csv
(collector and parser),withaname.ddosia
(collector and parser) andshadowserver.bgp
(just anothershadowerver
parser). -
[data sources] Changed the
shadowserver.ftp
parser's constant value of thename
event attribute to"ftp allow password wo ssl"
(previously it was"ftp, clear text pass"
). -
[data sources] Fixed a bug in the
abuse-ch.urlhaus-urls
collector: removed the (mistakenly kept) rigid limit on numbers of events being sent. -
[data sources] Removed the
malwarepatrol.malurl
collector. -
[portal, rest api, stream api, admin panel, data pipeline] Added a new feature: Ignore Lists. From now on, n6 administrators/operators can use Admin Panel to create and manage Ignore Lists, each identified by a unique label, with optional comment, flagged as active or not, and -- what is most interesting -- containing any number of Ignored IP Networks (note: bare IP addresses as also accepted; they are automatically converted to
.../32
networks). Then6filter
component will mark -- with theignored
event field set toTrue
-- each event that contains theaddress
field whose value is a non-empty list that contains only such dicts whoseip
item's value matches at least one Ignored IP Network belonging to any active Ignore List (note that Ignore Lists are active by default); any other events are marked withignored
set toFalse
. For non-privileged users (whose organizations havefull_access=False
in the Auth DB), results generated by Portal, REST API, Stream API/n6anonymizer
andn6counter
/n6notifier
(as well as by Test REST API and Test Stream API/n6stream_test_api_generator
) do not include events marked asignored
. On the other hand, for privileged users (whose organizations havefull_access=True
in the Auth DB), results generated by those n6 components do include also such events, but then all events contain theignored
field (set toTrue
orFalse
) -- except that Stream API/n6anonymizer
(and Test Stream API/n6stream_test_api_generator
) always treat all users as if they were non-privileged. Additionally, privileged users can filter results from REST API's/Portal API's by using a new query parameter:ignored
(Boolean). -
[portal, admin panel, docs] Added a new feature: Organization Agreements. It allows the administrators of an n6 instance to use Admin Panel to define optional terms/agreements which then can be accepted/rejected by the existing and new (future) users of n6, via Portal, on behalf of their organizations. The new feature is comprehensively documented.
-
[portal, admin panel] Enhanced the Edit organization settings form in the Portal frontend and the corresponding backend stuff as well as the related Admin Panel stuff -- to allow adding and/or removing users within the logger user's organization (actually: requesting n6 administrators to, respectively, add/re-activate and/or deactivate those users...).
-
[portal, rest api, stream api, data sources, data pipeline, event db, lib] The
name
event field (event attribute) is now coerced by the n6 data pipeline's machinery (namely, byn6lib.record_dict.RecordDict
...) to pure ASCII (by replacing each non-ASCII character with?
), and is, generally, required by all other parts of n6 to be pure ASCII... (However, when it comes to how events'id
are computed by parsers, efforts have been made to keep that unaffected by the coercion -- so that resultantid
values remain the same as previously for the same input values ofname
.) Events stored in the Event DB are now also expected to havename
(if present) already coerced that way. (See also the descriptions of the Event-DB-related changes below...) -
[portal, rest api, data pipeline, event db, lib] The
count
event field (event attribute) is no longer constrained to be less than 32768 (now its maximum value is 4294967295 which seems big enough for any practical purposes...). Therefore,n6aggregator
does not set thecount_actual
field anymore. (See also the descriptions of the Event-DB-related changes below...) -
[portal, rest api, data pipeline, event db, lib] Non-BMP Unicode characters (i.e., Unicode codepoints greater than 0xFFFF) are now properly supported (if present) in values of the
url
andtarget
event fields (attributes), i.e., now they can be reliably stored in the Event DB and retrieved from the Event DB. (Previously, that was broken, because the MariaDB's legacy max-3-bytes charsetutf8
was used for the Event DB. See also the descriptions of the Event-DB-related changes below...) -
[portal, rest api, event db, lib] Filtering the results by the
url
event field (attribute) -- by using theurl
orurl.sub
query parameter -- is now stricter in some ways, because the underlying MariaDB collation (for the Event DB's columnurl
in theevent
table) changed fromutf8_unicode_ci
toutf8mb4_bin
(in particular, nowurl
values are compared in a case-sensitive manner). -
[portal, rest api, event db, lib] Filtering the results by the
target
event field (attribute) may behave in a slightly different way, because the underlying MariaDB collation (for the Event DB's columntarget
in theevent
table) changed fromutf8_unicode_ci
toutf8mb4_unicode_520_ci
. -
[portal, rest api, event db, lib] The
modified
event field (attribute) is now required (i.e., guaranteed to be present in every event). See also the descriptions of the Event-DB-related changes below... -
[portal, rest api, data pipeline, auth db, lib] Implemented several performance enhancements/fixes and optimizations regarding retrieving and caching authorization data from the Auth DB (that is, concerning the stuff implemented in the
n6lib.auth_api
module and related modules; the addition of therecent_write_op_commit
Auth DB table, mentioned later, is also related to that). One of those enhancements is a new optional mechanism called pickle cache (see the related configuration options mentioned later...). -
[portal, rest api, lib]
n6lib.db_events.n6NormalizedData.like_query()
: fixed a bug causing injecting LIKE's wildcards when querying REST API or Portal API using query parametersurl.sub
/fqdn.sub
(SQL pattern injection). It was not a security problem, but it might cause that too large results (supersets of correct results) would be obtained for queries involving the affected parameters. -
[portal, lib]
n6lib.pyramid_commons.mfa_helpers
: fixed the value and the use ofMFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS
(previously namedMFA_CODE_MAX_ACCEPTABLE_AGE_IN_SECONDS
). Before the fix, if a Portal user successfully used an MFA code to log in, doing that "too early" but still within that MFA code's validity period (making use of the clock drift tolerance feature), it was then possible, for the same user, to successfully use the same MFA code once again, by doing that sufficiently late yet still within the same validity period. The crux of the bug was that the period of treating MFA codes as "already spent" was too short. (Note that the fixed bug does not look like a serious security flaw.) -
[portal] Applied many GUI/UX-related Portal fixes and enhancements... Among others, from now on, dates/times on the Incidents page are consistently processed/presented using UTC times; also, support for some additional search parameters have been added.
-
[stream api, auth db, lib] Since now, all new organization have Stream API enabled by default (as the default value of the
stream_api_enabled
field of then6lib.auth_db.models.Org
model is nowTrue
). -
[admin panel, lib] All editable fields in the Admin Panel accepting an IP network (in the CIDR notation) now also accept a bare IP address (which is automatically converted to a
.../32
network). What has actually been changed is the validation procedure for allip_network
fields defined inn6lib.auth_db.models...
. (To make that possible,n6sdk.data_spec.fields.IPv4NetField
, and all its subclasses, gained a new option:accept_bare_ip
-- of typebool
, specifiable as a keyword argument to the constructor or a subclass attribute, withFalse
as the default value). -
[admin panel, lib] Added a new column, Is Active, to the Admin Panel's User list view; the new column represents a newly added property of
n6lib.auth_db.models.User
:is_active
-- whose value is always a logical negation of the (already existing)User
model's fieldis_blocked
(representing theuser
Auth DB table's columnis_blocked
). -
[docs] The n6's documentation: added a new article: n6 REST API; significantly improved/updated two existing articles: n6 Stream API and Docker-Based Installation; applied a bunch of fixes, improvements and updates to other parts of the documentation.
-
[event db, lib] Made numerous changes to the schema and basic setup of the Event DB (see, in particular, the
etc/mysql/initdb/*.sql
files...). Namely: the MariaDB engine used for the Event DB is now RocksDB (rather than TokuDB); the general Event DB's character set and collation (that apply, among others, to thename
column in theevent
table...) are nowascii
andascii_general_ci
(rather than the legacy max-3-bytes charsetutf8
with the collationutf8_unicode_ci
), except that, in theevent
table, the character set and collation for theurl
column are nowutf8mb4
andutf8mb4_bin
, and the character set and collation for thetarget
column are nowutf8mb4
andutf8mb4_unicode_520_ci
; the order of the components of theevent
table's primary key is now:time
,ip
,id
(previously:id
,time
,ip
); theevent
table's columnsmodified
anddip
are nowNOT NULL
(in the case ofdip
, the value 0 means that there is no actual value; actually, for thedip
column, this convention has been used for a long time); theevent
table's columnsdport
andsport
are now of typeSMALLINT UNSIGNED
(previously:INTEGER
, which was an unnecessary waste of space); theevent
table's columncc
is now of typeCHAR(2)
(previously:VARCHAR(2)
); theevent
table's columncount
is now of typeINTEGER UNSIGNED
whose max. value is 4294967295 (previously:SMALLINT
with max. value 32767, which was far too small); several database indexes have been added/adjusted/removed; also, as a part of implementation of the aforementioned Ignore Lists feature, a new column has been added to theevent
table:ignored
, of typeBOOL
; apart from all that, several SQL variables are now consistently set to sensible values (max_allowed_packet
,sql_mode
,time_zone
)... What is important from the point of view of the administrators of an n6 instance is that a suitable migration of the whole production Event DB content needs to be performed (manually). -
[auth db, lib] Added, as a part of implementation of the aforementioned Ignore Lists feature, two new Auth DB tables:
ignore_list
andignored_ip_network
. Apart from that, added new Auth DB tables related to other features/mechanisms (also mentioned above...):agreement
,org_agreement_link
,org_config_update_request_user_addition_or_activation_request
,org_config_update_request_user_deactivation_request
,recent_write_op_commit
,registration_request_agreement_link
. Obviously, related model classes have been added as needed (seen6lib.auth_db.models
) and any necessary field validators have been implemented (seen6lib.auth_db.validators
). What is important from the point of view of the administrators of an n6 instance is that the Alembic migrations machinery needs to be used to update the schema of the production Auth DB (for the instructions how to do it, seeN6Lib/n6lib/auth_db/alembic/README.md
). -
[config, data pipeline] From now on, the
n6recorder
's configuration optionconnect_charset
(in the configuration sectionrecorder
) is expected to be set to the valueutf8mb4
(not to the valueutf8
anymore!) -- unless there are some special circumstances and you really now what you are doing, and why! -
[config, portal, rest api] From now on, the configuration option
sqlalchemy_event_db_connect_charset
(in REST API's and Portal API's*.ini
files) is expected to be set to the valueutf8mb4
(not to the valueutf8
anymore!) -- unless there are some special circumstances and you really now what you are doing, and why! -
[config, data sources] The collectors whose classes inherit (directly or indirectly) from
n6datasources.collectors.base.BaseDownloadingCollector
now support a new configuration option,download_timeout
, which can be set to customize HTTP(s) request timeouts. -
[data pipeline] Added a new auxiliary executable:
n6exchange_updater
-- to update Stream-API-related AMQP exchange declarations and bindings (adding and deleting them as appropriate), according to the relevant Stream API settings in Auth DB. (The implementation of the component resides in then6datapipeline.aux.exchange_updater
module.) -
[config, portal, rest api] New configuration options regarding certain performance improvement mechanisms can now be specified in the REST API's and Portal API's
*.ini
file (see theauth api prefetching configuration
part of the relevant configuration prototype files). In particular, the aforementioned optional mechanism called pickle cache can be activated (see the comments in the related configuration prototype files regarding the optionsauth_api_prefetching.pickle_cache_dir
andauth_api_prefetching.pickle_cache_signature_secret
; please, take seriously the security considerations those comments include...). -
[config, portal] A new configuration option,
session_cookie_sign_secret
, can now be specified in the Portal API's*.ini
file to explicitly set the secret key to be used to sign user session cookies (please, see the comments in the related configuration prototype files regarding that option...). By default, the option's value is empty, causing the legacy behavior (a new secret to sign session cookies being automatically generated on each start of the Portal API server application). Note: setting the option is necessary if the Portal API server application is run using multiple OS processes (not just threads), otherwise user sessions cannot be handled properly. -
[config, portal, rest api, broker auth api, admin panel, data sources, data pipeline] From now on, wherever in n6 an AMQP connection is established, authentication to RabbitMQ (the AMQP server) can be configured to be made using the PLAIN mechanism, i.e., with username and password (note: SSL-based EXTERNAL authentication with an X.509 client certificate is still possible, just no longer as the only option). To learn how to configure your n6 components to use the newly supported method of authentication, see respective comments in the relevant config prototype files: ad source/pipeline components --
00_global.conf
; ad input for collectors based onn6datasources.collectors.AMQPCollector
--60_amqp.conf
; ad logging usingn6lib.log_helpers.AMQPHandler
-- eitherlogging.conf
orproduction.ini
(the latter -- only ad Portal API and REST API). Note that on production systems, regardless of the authentication mechanism, you should use only connections secured with SSL (TLS). -
[config, portal, rest api, broker auth api, admin panel, data sources, data pipeline] From now on, in configuration files for any n6 components, any configuration options that specify filesystem paths or lists of filesystem paths are expected to be absolute paths, i.e., relative paths might no longer be accepted. Note that paths like
~/something
and~user/something
(intended to be expanded by replacing a~
/~user
marker with the user's home directory path) are still OK. -
[setup, lib, admin panel] Updated versions of some external dependencies (including some security-related cases...); also, added a few new dependencies.
-
[lib, portal, rest api, broker auth api, admin panel, data sources, data pipeline, config, cli, docs, etc/docker, tests] Made a bunch of various changes/enhancements (including backward incompatible ones) and additions to the code and documentation (many related to the features and changes mentioned above...), plus various fixes/cleanups, some refactoring, modernization and adjustments/updates (among other things, many changes to accommodate some of the major Event-DB-related changes described above; as well as certain temporary hacks to ease the transition process)... Also, many tests (with data/fixtures/helpers) have been added, enhanced, fixed, refactored, adjusted/updated... More or less the same can be said about many n6 components' configuration prototype files, and about some other configuration-or-Docker-related stuff...
-
[portal, setup, tests] Regarding the implementation of the n6 Portal's frontend (React-based TS/JS code and related resources, together with development tooling): made a bunch of additions, fixes/changes/enhancements as well as some refactoring/modernization, plus external package updates and additions... Among others, upgraded
Node
,React
andTypeScript
, and implemented a comprehensiveJest
-based test suite... -
[etc/docker, docs] Added Mailhog to the Docker-related stuff.
-
[lib] Removed some constants/classes/methods/attributes/functions, in particular:
n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_LEGACY_CHARSET
,n6lib.data_selection_tools.CondPredicateMaker.visit_RecItemParamCond()
(replaced withvisit_RecItemCond()
, mentioned below),n6lib.db_events.CustomInteger
,n6lib.db_events.JSONText
(replaced withJSONMediumText
, mentioned below),n6lib.db_events.n6ClientToEvent.__json__()
,n6lib.db_events.n6NormalizedData.to_raw_result_dict()
,n6lib.ldap_api_replacement.LdapAPIConnectionError
,n6lib.pyramid_commons.mfa_helpers.MFA_CODE_MAX_ACCEPTABLE_AGE_IN_SECONDS
, (replaced withMFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS
, mentioned below). -
[lib] Added a new module:
n6lib.file_helpers
(providing three utility classes:FileAccessor
,StampedFileAccessor
andSignedStampedFileAccessor
; and one utility function:as_path()
; see their docstrings for more information...). -
[lib, data sources] Added numerous constants/classes/methods/attributes/functions, in particular:
n6datasources.base.parsers.BaseParser.ignored_csv_raw_row_prefixes
(and overridden, as appropriate, in some subclasses ofBaseParser
...),n6lib.amqp_helpers.AMQPConnectionParamsError
,n6lib.amqp_helpers.GUEST_PASSWORD
,n6lib.amqp_helpers.GUEST_USERNAME
,n6lib.amqp_helpers.MIN_REQUIRED_PASSWORD_LENGTH
,n6lib.amqp_helpers.get_amqp_connection_params_dict_from_args.set_log_warning_func()
,n6lib.amqp_helpers.SimpleAMQPExchangeTool
,n6lib.auth_api.AuthAPI.get_ignore_lists_criteria_resolver()
,n6lib.auth_db.fields.HTTPAbsoluteURLField
,n6lib.auth_db.models.Agreement
,n6lib.auth_db.models.IgnoredIPNetwork
,n6lib.auth_db.models.IgnoreList
,n6lib.auth_db.models.Org.agreements
,n6lib.auth_db.models.org_agreement_link
,n6lib.auth_db.models.OrgConfigUpdateRequest.user_addition_or_activation_requests
,n6lib.auth_db.models.OrgConfigUpdateRequest.user_deactivation_requests
,n6lib.auth_db.models.OrgConfigUpdateRequestUserAdditionOrActivationRequest
,n6lib.auth_db.models.OrgConfigUpdateRequestUserDeactivationRequest
,n6lib.auth_db.models.RecentWriteOpCommit
,n6lib.auth_db.models.RegistrationRequest.agreements
,n6lib.auth_db.models.registration_request_agreement_link
,n6lib.auth_db.models.User.is_active
,n6lib.class_helpers.LackOf
,n6lib.common_helpers.ip_int_to_str()
(+n6sdk.addr_helpers.ip_int_to_str()
),n6lib.common_helpers.PY_NON_ASCII_ESCAPED_WITH_BACKSLASHREPLACE_HANDLER_REGEX
(+n6sdk.regexes.PY_NON_ASCII_ESCAPED_WITH_BACKSLASHREPLACE_HANDLER_REGEX
),n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_CONNECT_CHARSET_DEFAULT
,n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_SQL_MODE
,n6lib.data_selection_tools.IsTrueCond
,n6lib.data_selection_tools.CondBuilder.RecItemCondBuilder.is_true()
,n6lib.data_selection_tools.CondPredicateMaker.visit_RecItemCond()
,n6lib.data_spec.N6DataSpec.ignored
(a new event field specification),n6lib.db_events.JSONMediumText
,n6lib.db_events.n6NormalizedData.single_flag_query()
,n6lib.ldap_api_replacement.LdapAPI.peek_database_ver_and_timestamp()
,n6lib.pyramid_commons.mfa_helpers.DELAY_TO_BE_SURE_THAT_MFA_CODE_EXPIRES
,n6lib.pyramid_commons.mfa_helpers.MFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS
,n6lib.record_dict.N6DataSpecWithOptionalModified
,n6lib.record_dict.RecordDict.adjust_ignore()
,n6lib.sqlalchemy_related_test_helpers.sqlalchemy_type_to_str()
,n6lib.threaded_async.Future.peek_result()
.n6lib.typing_helpers.HashObj
.
4.5.0 (2023-11-29)
- [data sources, setup, config, etc/docker, tests] Globally renamed the
spamhaus.edrop
parser's classSpamhausEdrop202303Parser
(defined inn6datasources.parsers.spamhaus
and referred to in a few other places -- in particular, being the name of the-parser-dedicated configuration section!) toSpamhausEdropParser
, as well as the executable scriptn6parser_spamhausedrop202303
ton6parser_spamhausedrop
; also, fixedn6datasources.tests.parsers.test_spamhaus.TestSpamhausEdropParser
by removing its attributePARSER_RAW_FORMAT_VERSION_TAG
. The rationale for these changes is that no raw format version tag has ever been assigned to thespamhaus.edrop
parser.
-
[data pipeline, lib]
n6filter
: fixed a bug (in the machinery ofn6lib.auth_api.InsideCriteriaResolver
...) related to event ownership criteria (aka "inside" resource events criteria) regarding the very unlikely (yet not impossible) corner case of the0.0.0.0/32
IP network defined as such a criterion in the Auth DB... The bug might maken6filter
reject all incoming data (because of raised exceptions). -
[tests, docs] Non-major enhancements and fixes regarding some unit tests and documentation.
- [tests]
n6datasources.tests.parsers._parser_test_mixin
: enhanced certainParserTestMixin
-provided checks related to raw format version tags.
4.4.0 (2023-11-23)
-
[data sources, config] Added support for the
shadowserver.msmq
source (by adding the parser for it, as there already exists one common collector for allshadowserver.*
sources; obviously, appropriate additions have been made in the collector's and parser's sections in theN6DataSources/n6datasources/data/conf/60_shadowserver.conf
config prototype file). -
[data sources, config] Removed support for the following sources:
blueliv.map
anddarklist-de.bl
(removed both collectors and parsers!) as well asshadowserver.modbus
(removed just this source's parser). -
[data sources] The parsers for the
dataplane.*
sources have been changed to support the current data format (there was a need to change the delimiter and the row parsing mechanism...). -
[data sources] The collector for the
abuse-ch.ssl-blacklist
source (implemented inn6datasources.collectors.abuse_ch
as the class namedAbuseChSslBlacklistCollector
) used to be able to load the collector state in a legacy format related to the value of the class attributerow_time_legacy_state_key
-- that format is no longer supported, as the base class_BaseAbuseChDownloadingTimeOrderedRowsCollect
no longer makes use of that attribute. Note: these changes are relevant and breaking only if you need to load your collector state in that old format -- almost certainly you do not. -
[data sources] A new processing mechanism has been added to numerous existing parsers for
shadowserver.*
sources (by enhancing the_BaseShadowserverParser
class, defined in then6datasources.parsers.shadowserver
module) -- concerning events categorized as"amplifier"
. The mechanism is activated when aCVE-...
-like-regex-based match is found in thetag
field of the input data -- then the parser, apart from yielding an event (hereinafter referred to as a basic event) withcategory
set to"amplifier"
, also yields an extra event -- which is identical to the basic one, except that itscategory
is set to"vulnerable"
and itsname
is set to the regex-matched value (which is, basically, the CVE identifier). Because of that,name
andcategory
should no longer be declared as parser'sconstant_items
, so now_BaseShadowserverParser
provides support foradditional_standard_items
(which is a parser class's attribute similar toconstant_items
). For relevant parser classes, thename
andcategory
items have been moved from theirconstant_items
to theiradditional_standard_items
. -
[data sources] Now the generic
*.misp
collector supports loading state also in its legacy Python-2-specific format. -
[data sources, data pipeline, lib] A new restriction (implemented in
n6lib.data_spec.fields
, concerning theIPv4FieldForN6
andAddressFieldForN6
classes) is that, from now on, the zero IP address (0.0.0.0
) is neither a valid component IP within a record dict'saddress
(i.e., its items'ip
) orenriched
(i.e., keys in the mapping being its second item), nor a valid value of a record dict'sdip
. Note that this restriction regards all parsers and most of the other data pipeline components (via the machinery ofn6lib.record_dict.RecordDict
et consortes...). -
[data pipeline] The name of the AMQP input queue declared by
n6enrich
has been changed (!) fromenrichement
toenrichment
. -
[data pipeline] The
n6enrich
pipeline component (implemented inn6datapipeline.enrich
): from now on, the zero IP address (0.0.0.0
), irrespective of its exact formatting (i.e., regardless whether some octets are formatted with redundant leading zeros), is no longer taken into account when IPs are extracted fromurl
s, and whenfqdn
s are resolved to IPs. -
[data pipeline, event db, config] From now on, when
n6recorder
, during its activity (i.e., withinRecorder.input_callback()
...), encounters an exception which represents a database/DB API error (i.e., an instance of aMySQLdb.MySQLError
subclass, possibly wrapped in (an) SQLAlchemy-specific exception(s)...) whose error code (i.e.,<exception>.args[0]
being anint
, if any) indicates a fatal condition -- then aSystemExit(<appropriate message>)
is raised, so that the AMQP input message is requeued and then6recorder
executable script exits with a non-zero status. The set of error codes which are considered fatal (i.e. which trigger this behavior) is configurable -- by setting thefatal_db_api_error_codes
configuration option in therecorder
section; by default, that set includes only one value:1021
(i.e., theERR_DISK_FULL
code -- see the error codes listing on the MariaDB website). -
[portal, rest api, stream api, data pipeline, lib] A security-related behavioral fix has been applied to the event access rights and event ownership machinery (implemented in
n6lib.auth_api
...): from now on, IP-network-based access or ownership criteria (those stored in thecriteria_ip_network
andinside_filter_ip_network
Auth DB tables) referring to networks that contain the zero IP address (0.0.0.0
) are translated to IP address ranges whose lower bound is0.0.0.1
(in other words,0.0.0.0
is excluded). Thanks to that, events withoutip
are no longer erroneously considered as matching such IP-network-based criteria. In practice, from the security point of view, the fix is most important when it comes to Portal and REST API (considering that those components query the Event DB, in records of which the absence of an IP is, for certain technical reasons, represented by the value0
rather thanNULL
). For other involved components, i.e.,n6filter
andn6anonymizer
/Stream API, the security risk was rather small or non-existent. Note: as the fix is also related ton6filter
, it affects values ofmin_ip
in theinside_criteria
part of the JSON returned by the Portal API's endpoint/info/config
; they are displayed by the Portal's GUI: in the Account information page, in the "Inside" resource events criteria section, below the IP network filter label -- as IP ranges' lower bounds. -
[portal, rest api, lib] A behavioral fix related to the one described above (yet, this time, not related to security) has been applied to the procedure of translation of the
ip.net
request parameter to the corresponding fragment of Event DB queries (see: theip_net_query()
method ofn6lib.db_events.n6NormalizedData
...): from now on, each value that refers to a network which contains the zero IP address (0.0.0.0
) is translated to an IP address range whose lower bound is0.0.0.1
(in other words,0.0.0.0
is excluded); thanks to that, events with noip
are no longer erroneously included in such cases. -
[portal, rest api, lib] A new restriction (implemented in
n6lib.data_spec.fields
, concerning theIPv4FieldForN6
andAddressFieldForN6
classes) is that the zero IP address (0.0.0.0
) is no longer a valid value of theip
anddip
request parameters received by REST API's endpoints and analogous Portal API's endpoints. Also, regarding the Portal's GUI, the front-end validation part related to the IP search parameter has been appropriately adjusted. -
[portal, rest api, lib] The mechanism of result data cleaning (implemented as a part of a certain non-public stuff invoked in
n6lib.data_spec.N6DataSpec.clean_result_dict()
) has been enhanced in such a way that theaddress
field of cleaned result dicts no longer includes any items withip
equal to the zero IP address (0.0.0.0
), i.e., they are filtered out even if they appear in some Event DB records (they could when it comes to legacy data). Note that it is complemented by the already existing mechanism of removing from raw result dicts anyip
anddip
fields whose values are equal to the zero IP address (see:n6lib.db_events.make_raw_result_dict()
...). -
[rest api, config, lib]
n6lib.generate_test_events
: several changes and enhancements regarding theRandomEvent
class have been made, including backward incompatible additions/removals/modifications of options defined by its config spec, affecting the way the optional test REST API application (provided byn6web.main_test_api
et consortes...) is configured usinggenerator_rest_api.*
options... Also, most of theRandomEvent
's configuration-related stuff has been factored out to a new mixin class,RandomEventGeneratorConfigMixin
.
-
[data sources, data pipeline, config, etc/docker] Added, fixed, changed and removed several config prototype (
*.conf
) files in the directories:N6DataSources/n6datasources/data/conf/
,N6DataPipeline/n6datapipeline/data/conf/
andetc/n6/
. Note: for some of them, manual adjustments in user's actual configuration files are required (see the relevant comments in those files...). -
[setup, lib]
N6Lib
's dependencies: changed the version ofdnspython
from1.16
to2.4
. Also, added a new dependency,importlib_resources
, with version locked as>=5.12, <5.13
. -
[setup, data pipeline]
N6DataPipeline
's dependencies: temporarily locked the version ofintelmq
as<3.2
.
-
[data pipeline]
n6datapipeline.enrich.Enricher
: renamed theurl_to_fqdn_or_ip()
method tourl_to_hostname()
, and changed its interface regarding the return value: now it is always either a non-emptystr
orNone
. -
[lib]
n6lib.common_helpers
andn6sdk.encoding_helpers
: renamed thetry_to_normalize_surrogate_pairs_to_proper_codepoints()
function toreplace_surrogate_pairs_with_proper_codepoints()
. -
[lib] Removed three functions from
n6lib.common_helpers
:is_ipv4()
,is_pure_ascii()
andlower_if_pure_ascii()
. -
[lib]
n6lib.db_events
: removedIPAddress
's constant attributesNONE
andNONE_STR
(instead of them use then6lib.const
's constantsLACK_OF_IPv4_PLACEHOLDER_AS_INT
andLACK_OF_IPv4_PLACEHOLDER_AS_STR
). -
[lib]
n6lib.record_dict
: removedRecordDict
's constant attributesetitem_key_to_target_key
(together with some internal experimental mechanism based on it...). -
[lib]
n6lib.url_helpers
: changednormalize_url()
's signature and behavior... -
[tests]
n6datasources.tests.parsers._parser_test_mixin.ParserTestMixin
(and all inheriting parser test classes): added checking that if the parser'sdefault_binding_key
includes the raw format version tag segment then that segment matches the test class's attributePARSER_RAW_FORMAT_VERSION_TAG
.
-
[data sources] Added missing
re.ASCII
flag to regex definitions in a few parsers:sblam.spam
,spamhaus.drop
andspamhaus.edrop
(the lack of that flag caused that the affected regexes were too broad...). -
[data sources, config] Restored, in the
ShadowserverMailCollector
section of theN6DataSources/n6datasources/data/conf/60_shadowserver.conf
config prototype file, the (mistakenly deleted)"Poland Netcore/Netis Router Vulnerability Scan":"netis"
item of thesubject_to_channel
mapping. -
[data pipeline]
n6enrich
: fixed a few bugs concerning extraction of the hostname being a domain name (to becomefqdn
) or an IP address (to becomeip
inaddress
...) fromurl
. Those bugs caused that, for certain (rather uncommon) cases of malformed or untypical URLs, whole events were rejected (because of an exception), or (only for some cases and only if the Python's assertion-removal optimization mode was in effect) the resultant event'senriched
field erroneously included the"fqdn"
marker whereasfqdn
was not successfully extracted fromurl
. -
[data pipeline] Fixed
n6anonymizer
: now output bodies produced by the_get_result_dicts_and_output_body()
method ofn6datapipeline.aux.anonymizer.Anonymizer
are of the proper type (bytes
)... -
[admin panel] Fixed a RIPE search-related bug in the Admin Panel (in
N6AdminPanel/n6adminpanel/static/lookup_api_handler.js
-- in theRipePopupBase._getListsOfSeparatePersonOrOrgData()
function where the initial empty list was inadvertently added to theresultList
, leading to duplicate data entries in certain cases; this update ensures that a newcurrentList
is only added toresultList
upon encountering a valid separator and if it contains any data, preventing the addition of an empty initial list and the duplication of the first data set). -
[admin panel, lib] Extended the scope of data obtained from RIPE and displayed in the Admin Panel -- thanks to adding an
org
-key-based search feature to then6lib.ripe_api_client.RIPEApiClient
, which enables it to perform additional searches when encountering theorg
key; the enhancement allows for the retrieval and integration of organization-specific results into the existing data set (broadening the overall search capabilities). -
[etc/docker] Replaced expired test/example certificates.
-
[data sources, data pipeline, portal, setup, config, cli, lib, tests, etc/docker, docs] Various additions, fixes, changes, enhancements as well as some cleanups and code modernization/refactoring.
-
[lib]
n6lib.common_helpers
: from now on, theip_network_tuple_to_min_max_ip()
function (also available vian6sdk.encoding_helpers
) accepts an optional flag argument,force_min_ip_greater_than_zero
. -
[lib]
n6lib.common_helpers
: added theas_str_with_minimum_esc()
function (also available vian6sdk.encoding_helpers
). -
[lib]
n6lib.const
: added theLACK_OF_IPv4_PLACEHOLDER_AS_INT
(equal to0
) andLACK_OF_IPv4_PLACEHOLDER_AS_STR
(equal to"0.0.0.0"
) constants. -
[lib, tests]
n6lib.unit_test_helpers
: added toTestCaseMixin
a new helper method,raise_exc()
. -
[lib] Various additions, changes and removals regarding experimental code.
4.0.1 (2023-06-03)
- [docs, setup] Fixed generation of the docs by upgrading
mkdocs
to the version1.2.4
.
4.0.0 (2023-06-03)
This release is a big milestone.
Among others:
-
the n6 Portal gained support for OpenID Connect-based single sign-on (SSO) authentication;
-
the n6 Stream API (STOMP-based) now supports authentication based on API keys (those which have already been accepted by the n6 REST API); the new mechanism, implemented as a part of the
N6BrokerAuthApi
package, replaces the previously used mechanism (which was based on X.509 client certificates); -
added a significant number of components obtaining and processing security event data from external sources: 26 collectors and 86 parsers; now, in total, we have 35 collectors and 91 parsers (see the
N6DataSources
package); -
got rid of the Python-2-compatible legacy code (most of which were Python 2 versions of collectors and parsers) that used to reside in
N6Core
; the accompanying Python 2 packages (N6CoreLib
,N6Lib-py2
andN6SDK-py2
) have also been removed; note that the components related to active data sources have been migrated to Python 3 (8 collectors and 7 parsers -- now they reside inN6DataSources
); therefore, n6 is now Python-3-only (finally!); -
significant performance improvements have been accomplished: certain kinds of data queries (via the n6 REST API or n6 Portal) have become much faster, and
n6aggregator
's memory consumption has been considerably reduced; -
also, many minor improvements, a bunch of fixes, some refactoring and various cleanups have been made.
Note that some of the changes are not backwards compatible.
[...]
3.0.1 (2021-12-03)
-
[docs] A bunch of fixes and improvements regarding the documentation, including major changes to its structure, layout and styling.
-
[setup]
do_setup.py
: regarding the default value of the option--additional-packages
under Python 3, the version of themkdocs
package has been pinned (1.2.3
), and themkdocs-material
package (providing thematerial
docs theme) has been added (and its version is also pinned:8.0.3
); regarding the same under Python 2, themkdocs
package has been removed.
3.0.0 (2021-12-01)
This release is a big milestone. It includes, among others:
-
migration to Python 3
-
in the n6 data pipeline infrastructure: optional integration with IntelMQ
-
in the n6 Portal: a new frontend (implemented using React), two-factor authentication (based on TOTP), user's/organization's own data management (including config update and password reset forms, with related e-mail notices), and other goodies...
-
in the n6 REST API: API-key-based authentication
-
and many, many more improvements, a bunch of fixes, as well as some refactoring, removals and cleanups...
Note that many of the changes are not backwards compatible.
Also, note that most of the main elements of n6 -- namely:
N6DataPipeline
, N6DataSources
, N6Portal
, N6RestApi
,
N6AdminPanel
, N6BrokerAuthApi
, N6Lib
and N6SDK
-- are now
Python-3-only (more precisely: are compatible with CPython 3.9).
[...]
2.0.0 (2018-06-22)
The first public release of n6.