[ENH] Implementation of TS2Vec

[gasperpetelin]

The first approach for implementing TS2Vec is still a work in progress.

Usage:

from aeon.transformations.collection.contrastive_based import TS2Vec
X, _ = load_unit_test(split="train")

# Select the device
transformer = TS2Vec(device="cpu") # or
transformer = TS2Vec(device="cuda") # or
transformer = TS2Vec(device="cuda:0")

transformer.fit_transform(X)

Reference Issues/PRs

Implements TS2Vec as discussed in #2753

What does this implement/fix? Explain your changes.

Implements TS2Vec

Does your contribution introduce a new dependency? If yes, which one?

No new dependencies. The code is mostly based on Pytorch from original author

Any other comments?

The original code has MIT licence

PR checklist

For all contributions

• I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you after the PR has been merged.
• The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

For new estimators and functions

• I've added the estimator/function to the online API documentation.
• (OPTIONAL) I've added myself as a __maintainer__ at the top of relevant files and want to be contacted regarding its maintenance. Unmaintained files may be removed. This is for the full file, and you should not add yourself if you are just making minor changes or do not want to help maintain its contents.

For developers with write access

• (OPTIONAL) I've updated aeon's CODEOWNERS to receive notifications about future changes to these files.

[aeon-actions-bot]

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ $\color{#FEF1BE}{\textsf{enhancement}}$ ].
I have added the following labels to this PR based on the changes made: [ $\color{#2C2F20}{\textsf{testing}}$, $\color{#41A8F6}{\textsf{transformations}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

• Run pre-commit checks for all files
• Run mypy typecheck tests
• Run all pytest tests and configurations
• Run all notebook example tests
• Run numba-disabled codecov tests
• Stop automatic pre-commit fixes (always disabled for drafts)
• Disable numba cache loading
• Push an empty commit to re-run CI checks

[MatthewMiddlehurst]

Testing with soft dependencies also needs to be isolated i.e. https://github.com/aeon-toolkit/aeon/blob/main/aeon/classification/feature_based/tests/test_tsfresh.py

[gasperpetelin]

Thanks for pointing me in the correct direction. skipif was added for the torch dependency.

[TonyBagnall]

hi, thanks very much for this. I think we need to have a read of the paper, I'm not familiar myself, apologies if we are a little slow but if we introduce a new category of collection transformer we need to make sure we understand it. Are there other algorithms in this class that could follow on?

[TonyBagnall]

from my initial look, it seems that this transformer outputs a multivariate time series of embeddings, so input a collection of shape (n_cases, n_channels, n_timepoints) to outputs (n_cases, n_embeddings, n_timepoints), is that correct? If so, the output type is not tabular, it is a collection of transformed time series. Have I understood correctly?

Also, our estimators generally dont work with tensors. Could you format the output to by a 3D numpy array (n_cases, n_embeddings, n_timepoints),?

I'd like to try this for classification, could be interesting.

ah, having talked to hadi the collection output is an option for ts2vec, usual output is tabular.

[hadifawaz1999]

Waiting on the self supervised PR to get in then will review this, @TonyBagnall check the self supervised PR if you want to know more about how the structure of the module is gonna be for now (#2385 )

[gasperpetelin]

Hi. Just to clarify a few things about dimensions. I specifically fixed the TS2Vec implementation parameter to encoding_window='full_series', meaning that the output is always tabular with a shape of (n_cases, n_embeddings). Original TS2Vec implementation supports different encoding_window types, and one of them returns a shape of (n_cases, n_embeddings, n_timepoints), but I think for classification, (n_cases, n_embeddings) is the most relevant. In this regard, it is similar to TSFresh or Rocket.

Example:

from aeon.transformations.collection.contrastive_based import TS2Vec
from aeon.datasets import load_classification

for dataset in ['BasicMotions', 'Car']:
    print(dataset)
    X, y = load_classification(name=dataset) 
    transformer = TS2Vec(device="cuda", output_dim=320)
    print('\tinput:', X.shape, type(X))
    rtrn = transformer.fit_transform(X)
    print('\toutput:', rtrn.shape, type(rtrn))

Output:

BasicMotions
        input: (80, 6, 100) <class 'numpy.ndarray'>
        output: (80, 320) <class 'numpy.ndarray'>
Car
        input: (120, 1, 577) <class 'numpy.ndarray'>
        output: (120, 320) <class 'numpy.ndarray'>

When #2385 is merged, I can move the model into self_supervised module. As far as I know, contrastive models are just a subcategory of self-supervised ones.

[TonyBagnall]

hi @gasperpetelin #2385 is merged, it would be great if you could get this in

[gasperpetelin]

Hi @TonyBagnall. I’ve now moved my code to the self_supervised module. However, I’m not sure why some tests are failing. It looks like they are trying to connect to timeseriesclassification.com but are failing because the website is currently unreachable (not 100% sure if there is not another reason).

[hadifawaz1999]

Gonna make some general comments before starting to get into details with this:

• The setup we're following in self supervised is that the model should take a network as input parameter (see what TRILITE does) from the networks module of aeon (aeon/networks)
• The only thing the self supervised model should do is the training part, nothing related to the neural networks should be mentioned here.
• The same fit mechanism of TRILITE should be followed, meaning callbacks etc. (see fit function of TRILITE)
• Saving and loading of best model should be followed same as TRILITE
• The implementation for now is supporting tensorfow keras backend, when removing the network related stuff in the model implementaiton the fit function should look like the one in TRILITE, if the model uses a custom loss then it can be implemented as a private function of the class
• Most important thing is: keeping consistency with TRILITE

[gasperpetelin]

@hadifawaz1999 I just have 3 questions regarding the implementation:

• I’ve moved the TS2Vec neural network definition to aeon/networks. However, TS2Vec internally uses PyTorch, while all other networks in that directory are implemented using TensorFlow. This creates a potential issue: if a user passes a TS2Vec network to TRILITE, it would fail. The same would happen if someone passes a TensorFlow-based network like FCNNetwork to TS2Vec. So, if we're placing networks in the same module, should compatibility checks be implemented to prevent users from mixing incompatible components?

• Networks in aeon/networks currently inherit from BaseDeepLearningNetwork, which is designed for TensorFlow and expects specific methods, like returning the first and last layers. Since TS2Vec uses PyTorch, it can't inherit from this base class without making the PyTorch code somewhat hacky to conform to the expected interface. Should PyTorch code therefore be in aeon/networks?

• TensorFlow/Keras uses callbacks. To my knowledge, PyTorch does not have an equivalent standard (except PyTorch Lightning). Is it okay if the callback parameter simply accepts a regular Python function aka Callable (or a list of functions)?
  For example:

after_iter_callbacks: Callable | list[Callable] | None = None
after_epoch_callbacks: Callable | list[Callable] | None = None

[hadifawaz1999]

@hadifawaz1999 I just have 3 questions regarding the implementation:

* I’ve moved the TS2Vec neural network definition to `aeon/networks`. However, TS2Vec internally uses PyTorch, while all other networks in that directory are implemented using TensorFlow. This creates a potential issue: if a user passes a TS2Vec network to TRILITE, it would fail. The same would happen if someone passes a TensorFlow-based network like FCNNetwork to TS2Vec. So, if we're placing networks in the same module, should compatibility checks be implemented to prevent users from mixing incompatible components?

* Networks in `aeon/networks` currently inherit from `BaseDeepLearningNetwork`, which is designed for TensorFlow and expects specific methods, like returning the first and last layers. Since TS2Vec uses PyTorch, it can't inherit from this base class without making the PyTorch code somewhat hacky to conform to the expected interface. Should PyTorch code therefore be in `aeon/networks`?

* TensorFlow/Keras uses callbacks. To my knowledge, PyTorch does not have an equivalent standard (except PyTorch Lightning). Is it okay if the callback parameter simply accepts a regular Python function aka Callable (or a list of functions)?
  For example:

after_iter_callbacks: Callable | list[Callable] | None = None
after_epoch_callbacks: Callable | list[Callable] | None = None

The model should be re-implemented in tensorflow keras for the network and training strategy

[gasperpetelin]

Regarding the TensorFlow reimplementation: I’ll give it a try, but to be honest, I have significantly more experience with PyTorch than with TensorFlow/Keras. Because of that, implementing TS2Vec in TensorFlow is not something I can do easily or quickly.

TS2Vec has a non-trivial architecture and training procedure. Additionally, the two frameworks differ in several important aspects (for example, PyTorch's AveragedModel, handling of padding or NaNs, and weight initialization), which can lead to degraded performance even when the implementations appear almost identical. So while I’ll do my best, I can’t promise a fully equivalent TensorFlow version in the near future.

Just to confirm, TS2Vec would only be accepted as a TensorFlow/Keras implementation and not as a PyTorch version (similar to how HydraTransformer is handled)?

[hadifawaz1999]

Regarding the TensorFlow reimplementation: I’ll give it a try, but to be honest, I have significantly more experience with PyTorch than with TensorFlow/Keras. Because of that, implementing TS2Vec in TensorFlow is not something I can do easily or quickly.

TS2Vec has a non-trivial architecture and training procedure. Additionally, the two frameworks differ in several important aspects (for example, PyTorch's AveragedModel, handling of padding or NaNs, and weight initialization), which can lead to degraded performance even when the implementations appear almost identical. So while I’ll do my best, I can’t promise a fully equivalent TensorFlow version in the near future.

Just to confirm, TS2Vec would only be accepted as a TensorFlow/Keras implementation and not as a PyTorch version (similar to how HydraTransformer is handled)?

Yes the implementaiton should be in tensorflow-keras. The network itself should be easilly adapted, i dont really believe a model will be impacted by the initialization of the weights if you had to use another method, because if a model is impacted its really a bad sign on the model itself.

Padding exists in keras as well so there is no issue. Anything can be adapted easilly, there is no impossible way.

aeon supports the use of torch in other modules, but the deep learning framework no it follows a tensorflow backend and keras for the networks module