Kolmogorov Arnold Block for NBeats

[Sohaib-Ahmed21]

Description

Fixes: #1741

This PR adds Kolmogorov Arnold(KAN) Blocks in NBeats and also does refactoring of NBeats. Implementation of KAN blocks' layers is taken from original paper code.

Checklist

• Linked issues (if existing)
• Used pre-commit hooks when committing to ensure that code is compliant with hooks. Install hooks with pre-commit install.
  To run hooks independent of commit, execute pre-commit run --all-files

Make sure to have fun coding!

[Sohaib-Ahmed21]

@fkiraly @benHeid this PR is ready for review. Kindly review it. Thanks!

[Sohaib-Ahmed21]

@fkiraly @benHeid can you kindly review/merge it so I integrate NBEATSX modification in NBEATS without conflicts as I have asked @julian-fong and he is not working on NBEATSX.

[benHeid]

Not fully reviewed yet. Will continue in the next days. But I share my current comments so that you already receive some feedbacks.

When training KANs, the grid can be iteratively be refined. I wonder, if there is a way to implement this also here. However, this might probably more difficult and require changes to the trainer. So probably out of scope for this PR. Do you have opinions on that?

[benHeid]

The license of the original implementation is MIT. So in theory it is okay to copy the file. However, please add some credits at the top of the file.

Alternatively, we could think about adding KAN as a dependency.

@fkiraly do you have any additions on that matter?

[Sohaib-Ahmed21]

Okay, I will add the appropriate credits at the top of the file. Additionally, I agree that adding KAN as a dependency—perhaps as a soft dependency—seems like a good idea, especially considering its increasing relevance in time series forecasting.

[benHeid]

@fkiraly pinging you again to check if this is okay for you :)

[fkiraly]

which package are we exactly planning to add as a soft dep?

If it is a single layer, I think copying it over and including the license is perhaps better for now, because we do not have machinery to manage soft dependencies (like scikit-base or similar).

The proposed design in here sktime/enhancement-proposals#39 would allow that, but right now I think this would require a significant amounts of custom code to handle.

Or, is there an easy way that I am not seeing how the soft dependency import would work for part of the NN?

[Sohaib-Ahmed21]

which package are we exactly planning to add as a soft dep?

It is pykan library , reference https://pypi.org/project/pykan/

If it is a single layer, I think copying it over and including the license is perhaps better for now, because we do not have machinery to manage soft dependencies (like scikit-base or similar).

The proposed design in here sktime/enhancement-proposals#39 would allow that, but right now I think this would require a significant amounts of custom code to handle.

Or, is there an easy way that I am not seeing how the soft dependency import would work for part of the NN?

yes it is a single layer, only used in NBEATS. Also the library pykan has much more, but we only need this. I have implemented what you already suggested above. I have copied it over and included the license.

[fkiraly]

ok, makes sense, as long as the license points to the original source and is provided in full form (assuming it hsa the usual reqiurement to reproduce)

[Sohaib-Ahmed21]

Not fully reviewed yet. Will continue in the next days. But I share my current comments so that you already receive some feedbacks.

Thanks! Will address these reviews soon.

[Sohaib-Ahmed21]

When training KANs, the grid can be iteratively be refined. I wonder, if there is a way to implement this also here. However, this might probably more difficult and require changes to the trainer. So probably out of scope for this PR. Do you have opinions on that?

I'll explore this and share my thoughts.

[Sohaib-Ahmed21]

@benHeid I have addressed the reviews. Kindly review the updated PR.

When training KANs, the grid can be iteratively be refined. I wonder, if there is a way to implement this also here. However, this > might probably more difficult and require changes to the trainer. So probably out of scope for this PR. Do you have opinions on > that?

To address this, I have taken logic from original implementation of pykan library and made custom Callback i.e. GridUpdateCallback which updates grid after specified steps. This has been tested and working fine with improved results. Your thoughts will be helpful in this regard.

[Sohaib-Ahmed21]

@benHeid @fkiraly I have addressed the reviews. Kindly review it so I proceed integrating NBEATSX without conflicts.

[benHeid]

I only have one last comment. But I also would like to hear @fkiraly opinion on the license of the kan_layer

[fkiraly]

I do have one major request though: This PR seems to be changing the parametrization of NBEATS in major ways, and it essentially alters the structure.

May I hence request to, instead of modifying the existing class, to add a new class?
There may be common code that could be refactored, I just think that it is better to keep the class NBEATS - a common method - as a conceptual reference to the original NBEATS algorithm.

Variants and modifications I would provide as new classes - there are many modifications of NBEATS; so it seems like a suboptimal design decision to merge one of those (albeit a popular one) in the reference implementation.

[Sohaib-Ahmed21]

I do have one major request though: This PR seems to be changing the parametrization of NBEATS in major ways, and it essentially alters the structure.

May I hence request to, instead of modifying the existing class, to add a new class? There may be common code that could be refactored, I just think that it is better to keep the class NBEATS - a common method - as a conceptual reference to the original NBEATS algorithm.

Variants and modifications I would provide as new classes - there are many modifications of NBEATS; so it seems like a suboptimal design decision to merge one of those (albeit a popular one) in the reference implementation.

Sounds good, thanks for the review! I will make separate class for NBEATS using KAN keeping the original algorithm separate and will update the PR.

[fkiraly]

FYI @benHeid, thoughts/opinions?

[benHeid]

I see your point. However, I fear that we will introduce a lot of duplicated code. Thus, we might think about refactoring strategies.

But I am also happy with a new class, since this reduces also the complexity.

[Sohaib-Ahmed21]

I see your point. However, I fear that we will introduce a lot of duplicated code. Thus, we might think about refactoring strategies.

But I am also happy with a new class, since this reduces also the complexity.

I'll try to reuse as much code as possible like submodules, etc. So should I move towards new class implementation?

Update: Adding new class and will update PR soon.