Releases: databento/dbn
Releases · databento/dbn
0.14.0
Release notes
Enhancements
- This version begins the transition to DBN version 2 (DBNv2). In this version, the
decoders support decoding both versions of DBN and the DBN encoders default to
keeping version of the input. However, in a future version, decoders will by default
convert DBNv1 to DBNv2 and support will be dropped for encoding DBNv1.- Affects
SymbolMappingMsg
,InstrumentDefMsg
, andMetadata
. All other record
types and market data schemas are unchanged - Version 1 structs can be converted to version 2 structs with the
From
trait
- Affects
- Added
symbol_cstr_len
field toMetadata
to indicate the length of fixed symbol
strings - Added
stype_in
andstype_out
fields toSymbolMappingMsg
to provide more context
with live symbology updates - Added smart wrapping to
dbn
CLI help output - Updated
rtype_dispatch
family of macros to check record length to handle both
versions of records. This is temporary during the transition period - Added
VersionUpgradePolicy
enum and associated methods to the decoders to
allow specifying how to handle decoding records from prior DBN versions - Added
Metadata::upgrade()
method to updateMetadata
from a prior DBN version to
the latest version - Added
-u
/--upgrade
flags todbn
CLI that when passed upgrades DBN data from
previous versions. By default data is decoded as-is - Made
AsyncDbnDecoder::decode_record
,AsyncDbnDecoder::decode_record_ref
,
dbn::AsyncRecordDecoder::decode
, anddbn::AsyncRecordDecoder::decode_ref
cancellation safe. This makes them safe to use within a
tokio::select!
(https://docs.rs/tokio/latest/tokio/macro.select.html) statement - Added documention around cancellation safety for async APIs
- Improved error messages for conversion errors
- Added
TOB
flag to denote top-of-book messages - Added new publisher values in preparation for IFEU.IMPACT and NDEX.IMPACT datasets
- Added new publisher values for consolidated DBEQ.BASIC and DBEQ.PLUS
- Added
MAX_RECORD_LEN
constant for the length of the largest record type - Exposed record flag constants in
databento_dbn
withF_
prefix - Added export to Python for
RType
Breaking changes
- The old
InstrumentDefMsg
is nowcompat::InstrumentDefMsgV1
compat::InstrumentDefMsgV2
is now an alias forInstrumentDefMsg
- The old
SymbolMappingMsg
is nowcompat::SymbolMappingMsgV1
compat::SymbolMappingMsgV2
is now an alias forSymbolMappingMsg
- Changed
SYMBOL_CSTR_LEN
constant to 71. Previous value is now in
compat::SYMBOL_CSTR_V1
- Changed
DBN_VERSION
constant to 2 security_update_action
was converted to a rawc_char
to safely support adding
variants in the future- Renamed
_dummy
inInstrumentDefMsg
to_reserved
- Removed
_reserved2
,_reserved3
, and_reserved5
fromInstrumentDefMsg
- Removed
_dummy
fromSymbolMappingMsg
- Moved position of
strike_price
withinInstrumentDefMsg
but left text serialization
order unchanged - Made
Error
non-exhaustive, meaning it can no longer be exhaustively matched against.
This allows adding additional error variants in the future without a breaking change - Added
upgrade_policy
parameter toRecordDecoder::with_version
constructor to
control whether records of previous versions will be upgraded - Added
upgrade_policy
parameter toDynDecoder
constructors to control whether
records of previous versions will be upgraded - Renamed
symbol_map
parameter for Python Transcoder tosymbol_interval_map
to
better reflect the date intervals it contains
Deprecations
- Deprecated unused
write_dbn_file
function from Python interface. Please use
Transcoder
instead
Bug fixes
- Fixed typo in Python type definition for
InstrumentDefMsg.pretty_high_limit_price
- Fixed type signature for
Metadata.stype_in
andMetadata.stype_out
Python methods - Fixed incorrect version in
pyproject.toml
0.13.0
Release notes
Enhancements
- Added
SymbolMappingMsgV2::new
method - Added
Record
trait for all types beginning with aRecordHeader
- Added new
index_ts
andraw_index_ts
methods toRecord
trait, which returns the
primary timestamp for a record
- Added new
- Added
RecordMut
trait for accessing a mutable reference to aRecordHeader
- Implemented
PartialOrd
for all record types, based onraw_index_ts
- Loosened
DbnEncodable
from requiringHasRType
to only requiringRecord
. This means
RecordRef
s and concrete records can be encoded with the same methods
Breaking changes
- Split part of
HasRType
into newRecord
andRecordMut
traits, which are object-
safe: they can be used inBox<dyn>
.RecordRef
also implementsRecord
, so it's
easier to write code that works for both concrete records as well asRecordRef
- Removed
RecordRef
methods made redundant by it implementingRecord
- Removed
input_compression
parameter from PythonTranscoder
Deprecations
- Deprecated
SymbolIndex::get_for_rec_ref
, which was made redundant by loosening the
trait bound onSymbolIndex::get_for_rec
to acceptRecordRef
s
Bug fixes
- Fixed
TsSymbolMap
not always using the correct timestamp for getting the mapped
symbol
0.12.0
Release notes
Enhancements
- Added
map_symbols
support to PythonTranscoder
- Added new publisher variants in preparation for DBEQ.PLUS dataset
- Added
from_dataset_venue
function toPublisher
to facilitate destructuring - Implemented
Default
for most records to make testing easier - Added
from_zstd
function toAsyncDbnEncoder
to match synchronous encoder - Added re-exports for
enums::flags
,enums::rtype
,record::BidAskPair
,
record::RecordHeader
, andrecord::WithTsOut
to simplify imports - Added
--fragment
CLI flag for writing DBN without the metadata header - Added
--input-dbn-version
CLI option for specifying the DBN version of a DBN
fragment - Added
serde::Deserialize
implementations forDataset
,Venue
, andPublisher
- Added support for Python 3.12 to
databento_dbn
- Added
RecordDecoder::with_version
for future use when dealing with compatibility
between different DBN versions - Added new dispatch macros:
rtype_ts_out_method_dispatch
,
rtype_ts_out_async_method_dispatch
,rtype_method_dispatch
, and
schema_ts_out_method_dispatch
- Added
InstrumentDefMsgV2
andSymbolMappingMsgV2
for forward compatibility with a
version of DBN - Added
TsSymbolMap
andPitSymbolMap
to aid with both historical and live symbology- Added support for inverse symbology, i.e. with
stype_in=InstrumentId
- Added support for inverse symbology, i.e. with
Breaking changes
- Changed
Metadata::symbol_map
to returnTsSymbolMap
- Changed
Metadata::symbol_map_for_date
to returnPitSymbolMap
- Changed
Default
implementation forBidAskPair
by setting prices toUNDEF_PRICE
- Added new publisher values in preparation for DBEQ.PLUS
- Added
ts_out
parameter toencode_header_for_schema
inCsvEncoder
and
DynEncoder
to allow controlling whether "ts_out" is in the header - Added
from_dataset_venue
function toPublisher
to facilitate
destructuring.
0.11.1
Release notes
Enhancements
- Upgraded
async-compression
to 0.4.3 - Upgraded
csv
to 1.3 - Upgraded
num_enum
to 0.7
Bug fixes
- Changed DBN stream detection to ignore the DBN version
0.11.0
Release notes
Enhancements
- Added new
EncodeRecordTextExt
trait which is implemented for the CSV and JSON
encoders. It adds two methods for encoding asymbol
field along side the rest of the
record fields, matching the behavior ofmap_symbols
in the historical API - Added
encode_header
andencode_header_for_schema
methods toCsvEncoder
and
DynEncoder
to give more flexibility for encoding CSV headers - Added
from_file
andfrom_zstd_file
functions toAsyncDbnDecoder
to match
synchronous decoder - Implemented
Copy
forRecordRef
to make it behave more like a reference - Added
AsyncDbnEncoder
for simpler DBN encoding and to match sync API - Added
RecordEnum
andRecordRefEnum
to more easily be able to pattern match on
records of different types - Added
ARCX.PILLAR.ARCX
publisher - Added
From
DBN records forRecordRef
- Added re-exports to the top level of the crate for all enums and records for simpler
imports - Added
ClosePrice
andNetChange
StatType
s used in theOPRA.PILLAR
dataset
Breaking changes
- Split
encode_record_ref
into a safe method with no arguments and an unsafe method
with ats_out
parameter to reduceunsafe
usage when not working with live data
that may containts_out
Bug fixes
- Fixed
dbn
CLI not writing CSV header when using--fragment
and--zstd-fragment
flags - Fixed lifetime on return value from
RecordRef::get_unchecked
- Fixed missing check for
stype_out
before buildingMetadata
symbology maps
0.10.2
Release notes
Bug fixes
- Fixed query range checking in
Metadata::symbol_map_for_date
- Added
debug_assert_eq!
check for alignment inRecordRef::new
0.10.1
Release notes
Bug fixes
- Changed
Metadata::symbol_map
andsymbol_map_for_date
to returnString
values
instead of&str
, which made it difficult to use
0.10.0
Release notes
Enhancements
- Added
start
andend
getters toMetadata
that returntime::OffsetDateTime
- Added
symbol_map
andsymbol_map_for_date
methods toMetadata
to aid historical
symbology mapping from the instrument IDs in records - Added
DynReader
struct for being agnostic about whether an input stream is
zstd-compressed - Improved safety of
RecordRef::get
by adding length check - Added Python DBN
Transcoder
class for converting DBN to JSON and CSV with optional
zstd compression - Added optional
has_metadata
parameter to PythonDBNDecoder
to allow
decoding plain records by passingFalse
. By defaultDBNDecoder
expects a complete
DBN stream, which begins with metadata - Added
get_ref
methods todbn::Decoder
anddbn::RecordDecoder
which return a
reference to the inner reader - Added
UNDEF_PRICE
,UNDEF_ORDER_SIZE
,UNDEF_STAT_QUANTITY
, andUNDEF_TIMESTAMP
constants todatabento_dbn
Python package to make it easier to filter null values - Added
Metadata::builder()
function to create a new builder instance
Breaking changes
- Split out
EncodeRecordRef
trait fromEncodeDbn
to have a boxable trait (i.e.
Box<dyn EncodeRecordRef>
) for dynamic encoding - Split out
EncodeRecord
trait fromEncodeDbn
- Split out
DecodeRecordRef
trait fromDecodeDbn
to have a boxable trait (i.e.
Box<dyn DecodeRecordRef>
) for dynamic decoding - Changed
DynWriter
from an enum to a struct with only private fields
Bug fixes
- Fixed typo in
BATY.PITCH.BATY
publisher - Fixed typo in
README.md
(credit: @thomas-k-cameron)
0.9.0
Release notes
Enhancements
- Added
publisher
method toRecordHeader
and all record types for converting
thepublisher_id
to an enum - Added getters that return
time::OffsetDateTime
for the following fields:
ts_event
,ts_recv
,ts_ref
,activation
,expiration
,start_ts
,end_ts
,
ts_out
- Added getters for
ts_in_delta
that returntime::Duration
0.8.3
Release notes
Bug fixes
- Fixed missing
raw_instrument_id
field in PythonInstrumentDefMsg
- Fixed missing
OHLCV_EOD
variant in PythonSchema
type hint