Babeltrace 2.0.0 - Release Candidate 1

Hi everyone!

We’re very excited to announce the first Babeltrace 2.0 release candidate! 😃

What’s new since Babeltrace 2.0.0-pre5?

Babeltrace 2.0.0-pre5 was released 3 May 2019.

We brought many improvements to the CLI, the project plugins, and the library and its Python bindings.

We also updated all the Babeltrace 2 manual pages.

The build-time and run-time requirements for Babeltrace 2.0.0-rc1 are updated:

• Babeltrace 2.0.0-rc1 does not need the popt library anymore to parse command-line arguments.

  We wrote our own command-line argument parser to remove this obsolete requirement and to support advanced, position-dependent parsing.

• Babeltrace 2.0.0-rc1 requires GLib ≥ 2.28 at build time and run time instead of the previous ≥ 2.22 requirement.

Command-line interface

Automatic source component discovery

With the convert command’s new automatic source component discovery feature, specify a path, URI, or arbitrary string to let babeltrace2 find the most appropriate components to use:

$ babeltrace2 /path/to/ctf/trace
$ babeltrace2 net://localhost/host/myhost/my-session

This means you don’t need to create an explicit component with the --component option most of the time.

Behind the scenes, this feature queries the babeltrace.support-info object from all known component classes: a component class can reply with a confidence score to indicate its ability to handle a given path or custom string.

For example, a source.ctf.fs (CTF trace directory source) component class replies to this query whether or not the given path looks like a CTF trace directory.

NOTE: Source component classes do not have to implement the babeltrace.support-info query.

If a component class does not implement this query, the automatic source component discovery algorithm won’t consider it. In that case, you must instantiate such a component class explicitly with the --component option.

Plugins

New sink.text.details component class

This new component pretty-prints all the details of the messages it consumes.

The output is optimized to be human-readable, yet it is deterministic for testing and debugging purposes.

You can use a sink.text.details component to easily inspect messages flowing at any point of a graph and make it easy to test and troubleshoot other component classes.

The Babeltrace 2 project uses this component class for many tests.

source.ctf.fs

A source.ctf.fs component can now open and read malformed CTF traces produced by tracers having known bugs and corner cases (LTTng-UST, LTTng-modules, and barectf).

For example, a source.ctf.fs component can now read data streams which the lttng-crash tool generates, even if the last packets can be malformed.

See the Trace quirks section of the babeltrace2-source.ctf.fs(7) manual page for more details.

Library and Python bindings

Library-specific: simple sink component

Add a sink component to a graph easily using the new simple sink component C API.

A simple sink component is a sink component with a single input port and a single message iterator.

This interface reduces the amount of code you need to write to create a basic Babeltrace 2 message consumer.

Just provide a consumption user function which receives the component’s message iterator to consume upstream messages.

You can also implement other user functions:

Initialization function

Called once during the initialization phase of the graph.

Finalization function

Called once during the finalization phase of the graph.

Python-specific: automatic source component discovery using bt2.TraceCollectionMessageIterator

Integrate the same automatic source component discovery mechanism which the babeltrace2 CLI’s convert command uses to a Python application using a bt2.TraceCollectionMessageIterator object.

With a bt2.TraceCollectionMessageIterator object, your application can access additional component classes installed on the system from the get go.

New field class types

New field classes are available in libbabeltrace2 and its Python bindings to support eventual, new CTF 2 field classes.

The new fields are:

Boolean

Contains a boolean value.

Bit array

Contains a fixed-size array of bits.

A bit array field can represent a CPU status or flag register value, for example.

Unlike integer fields, a bit array field is not a numerical field.

Option

Contains an optional field.

An option field class is conceptually equivalent to a variant field class with two options: no field class and the optional field class.

User attributes

Attach custom user attributes to any of the following trace IR objects:

• Clock class

• Event class

• Field class

• Stream

• Stream class

• Trace

• Trace class

This new property exists because we expect the CTF 2 metadata objects to contain user attributes too.

Error reporting

Report rich error causes from user functions with the new thread-safe error reporting API.

The error reporting API makes it possible for the various actors interacting through libbabeltrace2 to precisely describe the chain of events that lead to a given error. As a library user, you can access this list of causes and reduce the time needed to troubleshoot problems.

This feature is similar to the stack trace which exception objects contain in many programming languages.

When an error occurs, the babeltrace2 CLI uses this error reporting API to show the top-level error itself, but also a detailed list of its causes, for example:

ERROR:    [Babeltrace CLI] (babeltrace2.c:2546)
  Graph failed to complete successfully
CAUSED BY [Babeltrace library] (graph.c:604)
  Component's "consume" method failed: ..., comp-name="sink.text.details", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-class-name="debug-info", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-name="muxer", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-class-name="MyFirstSource", ...
CAUSED BY [src.demo.MyFirstSource (some-name): 'source.demo.MyFirstSource'] (bt2/native_bt_log_and_append_error.h:100)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/dist-packages/bt2/message_iterator.py", line 151, in _bt_next_from_native
      msg = next(self)
    File "./bt_plugin_foo.py", line 33, in __next__
      raise Exception
  Exception

In the Python bindings, raise any exception object as usual from a user function: the bt2 package converts it to a libbabeltrace2 error cause for other parties.

Coinstallation with Babeltrace 1

To make the transition from Babeltrace 1 to Babeltrace 2 easier, we decided to make both projects coinstallable thanks to those changes:

• The Babeltrace 2 CLI tool is named babeltrace2.

• The Babeltrace 2 library’s include directory is named babeltrace2.

• The Babeltrace 2 library is named libbabeltrace2.

• The Babeltrace 2 manual page names start with babeltrace2-.

Distribution packages will be available for both Babeltrace 1 and Babeltrace 2.

Message Interchange Protocol versioning

Future-proof component classes using Message Interchange Protocol versioning.

As Babeltrace 2 evolves, we expect to introduce new concepts and objects to libbabeltrace2 such as field class types, message types, methods, and message ordering requirements. To ensure forward and backward compatibility can be maintained across future releases of Babeltrace 2, Babeltrace 2.0.0-rc1 adds the concept of Message Interchange Protocol (MIP).

Should we introduce a breaking change to the protocol which trace processing graph components must honor, we’ll bump the MIP version to announce the modified interface.

You can consider the MIP version as an implicit precondition variable for almost all the libbabeltrace2 functions.

Given the expected initialization parameters, a component class can report a range of supported MIP versions. You can then build a graph which operates with a specific MIP version to ensure all its components exchange messages using the same protocol.

const correctness

The Babeltrace 2 C API uses the const qualifier liberally to catch many programming errors at compile time.

As the Python language does not have a const feature to express immutability co...

v2.0.0-pre5

What's new since pre4?

New features

Trace merging

It is now possible to merge CTF traces that share the same UUID to a
single CTF trace. This is especially useful when combined with the
tracing session rotation feature introduced by LTTng 2.11.

For instance, it is possible to combine an arbitrary number of trace
archives and merge them back together.

Example:

A tracing session configured to automatically rotate every two seconds
will result in a large number of independent traces, as follows:

/path/to/my_session-20190501-181356
└──  archives
    ├──  20190501T181356-0400-20190501T181358-0400-1
    │   └──  ust
    │       └──  uid
    │           └──  1000
    │               └──  64-bit
    │                   ├──  channel0_0
    │                   ├──  channel0_1
    │                   ├──  channel0_2
    │                   ├──  channel0_3
    │                   ├──  index
    │                   └──  metadata
    ├──  20190501T181358-0400-20190501T181400-0400-2
    │   └──  ust
    │       └──  uid
    │           └──  1000
    │               └──  64-bit
    │                   ├──  channel0_0
    │                   ├──  channel0_1
    │                   ├──  channel0_2
    │                   ├──  channel0_3
    │                   ├──  index
    │                   └──  metadata
    ├──  20190501T181400-0400-20190501T181402-0400-3
    │   └──  ust
    │       └──  uid
    │           └──  1000
    │               └──  64-bit
    │                   ├──  channel0_0
    │                   ├──  channel0_1
    │                   ├──  channel0_2
    │                   ├──  channel0_3
    │                   ├──  index
    │                   └──  metadata
    ...

With this feature, it is now possible to combine all, or a subset, of
the archived trace chunks. The following example combines two of the
archived trace chunks into a single trace.

babeltrace [...]/archives/20190501T181358-0400-20190501T181400-0400-2 \
           [...]archives/20190501T181402-0400-20190501T181404-0400-4 \
           --output-format=ctf --output /tmp/merged-trace

This command produces the following trace:

/tmp/merged-trace
└──  myhostname
    └──  my_session-20190501T181356-0400
        └──  ust
            └──  uid
                └──  1000
                    └──  64-bit
                        ├──  channel0_0
                        ├──  channel0_1
                        ├──  channel0_2
                        ├──  channel0_3
                        └──  metadata

Note that this command uses new trace environment fields which will be
produced by the final release of LTTng 2.11. In the meantime, using
the existing LTTng 2.11-rc1 release will result in a different output
trace hierarchy.

LTTng live "subscribe" mode

It is now possible to configure the src.ctf.lttng-live component in
"subscribe" mode. In this mode, the source will attempt to consume an
LTTng relay daemon's session and retry periodically if the session
does not exist.

This is in contrast with the default behaviour which will return (with
success) immediately if the session to consume does not exist on the
LTTng relay daemon.

The following command demonstrates the use of this new mode:

babeltrace --component=src.ctf.lttng-live \
           --url=net://relayhost/host/tgthost/my-session \
           --params='session-not-found-action="continue"'

Note that we are planning the addition of a dedicated option to the CLI by the
time of the final release.

CLI support for array component parameters (--params)

Users of the CLI can now use array parameters when configuring a
component. See the following example:

babeltrace --component=src.my.class \
           --params='tokens=[23, 17, [42, "meow"]], path="/path/to/trace"'

CTF source can now recursively search paths (and path arrays)

Using the feature above, it is now possible to provide a list of paths
to the src.ctf.fs component and rely on the component to recursively
open CTF traces it finds.

See the following example:

babeltrace --component=src.ctf.fs \
           --params='paths=["trace1", "trace2", "trace3"]'

Performance improvements

One of the primary focus areas of the pre5 release was optimization
work to bring Babeltrace 2's CTF reader back to performance parity
with Babeltrace 1.x and reduce the overhead of the graph processing
infrastructure. As of this release, our benchmarks indicate that
Babeltrace 2 is now slightly faster (~4%) than the latest Babeltrace 1
release with a dummy output.

This work (and performance comparison) was the subject of a talk by
Philippe Proulx at the latest Tracing Summit. The video [1] and slides
[2] of this presentation are available and provide the rationale
behind the biggest changes to the API.

Beyond API changes, this work resulted in the addition of a new
"developer mode". When Babeltrace 2 is built in this mode, a large
number of checks are added to enforce most preconditions required by
the API.

This makes it easier to develop and test new plugins while ensuring
that they honour the contract of the various functions they use. Since
components are now assumed to have been tested in this mode by their
authors, the release configuration of Babeltrace 2 can eschew most
precondition checks at run time, resulting in improved performance.

To build Babeltrace 2 in this mode and test your own plugins, set the
BABELTRACE_DEV_MODE environment variable 1 at configure time, for
example:

BABELTRACE_DEV_MODE=1 ./configure

Moreover, you can use the BABELTRACE_MINIMAL_LOG_LEVEL variable to
compile out all logging below a given level (VERBOSE, DEBUG,
INFO, WARN, ERROR, and FATAL), resulting in further
performance improvements. The performance impact of enabling a logging
level of INFO (and up) is minimal as such statements are seldom used
in hot paths.

The current default logging level is left at VERBOSE and will remain
so until the final release. This setting incurs a performance penalty,
but will make it easier for users of this pre-release to report bugs
with detailed logs.

API changes

The performance improvement work has required significant API changes.
These changes were intrusive and have made it difficult to make
intermediary releases available, which is the main reason for the long
time between releases pre4 and pre5.

The APIs introduced by Babeltrace 2 made use of systematic reference
counting which was identified as a key culprit behind the performance
gap between Babeltrace 1.x and 2.x. A number of APIs routinely used in
the "hot paths" of trace processing now allow references to be
borrowed at no performance cost.

Among the biggest changes is the split between the CTF IR API
(previously introduced by the CTF writer library in Babeltrace 1.x)
and trace IR.

Trace IR is a higher-level trace access API that decouples the trace
processing graph from trace format implementation details. Since it is
decoupled from the CTF specification, we took the opportunity to adopt
a more generic (and sometimes clearer) terminology.

Observations made during the development of the new features of this
release and comments by users who have started using the pre4 API have
led us to adding "self" versions of objects which make use of
compile-time type checking to ensure that using functions that should
only be used in a given context results in warnings when they are
misused.

Moreover, const-related changes to the API will make it easier for
plugin authors to write const-correct code and provide an indication
when certain operations should not be performed.

Finally, an optional message iterator seeking API has been added to
allow seeking to a trace's beginning or to an arbitrary point
expressed in nanoseconds since the trace's origin.

We expect the API to undergo some minor changes before the first
release candidate. Among them figure:

• Addition of ranges associated to variant field classes.
• Introduction of a dependency link between upstream and
  downstream message iterators.
• Introduction of canceler objects, allowing long graph operations to be
  canceled (mainly meant for interactive use cases).
• Introduction of new types and user attributes in preparation for
  CTF 2.
• Versioning API to allow plugins to query graph capabilities.
• Introduction of unsigned/signed value objects.

A number of users had reported having started implementing plugins
using the pre4 API. Hence, we welcome any feedback regarding the API
changes.

Co-installation with Babeltrace 1.x

In order to facilitate the transition from Babeltrace 1.x to 2.x, we
will be making it easier to co-install both release series of
Babeltrace. To do so, the following changes are upcoming planned:

• Rename the CLI tool to babeltrace2. It is planned that
  packagers will symlink it as babeltrace when preferred, as
  is done for CPython packages.
• Rename the include directory to babeltrace2.
• Rename the library to libbabeltrace2.

Since co-installing both series will be possible, we plan to:

• Drop support for the babeltrace Python package (superseded by bt2).
• Drop support for the babeltrace-log util (you can do the
  equivalent with the Babeltrace 2 CLI).

Bug fixes

A large number of bug fixes are introduced as part of this release,
notably:

• Network efficiency improvements of the src.ctf.lttng-live
  component class.
• Fixes to the argument parsing of the CLI.
• CTF writer correctness fixes.
• Miscellaneous bug fixes in the various plugins.

Road to the final release

Here is a non-exhaustive list of changes planned before we ...