✨ Add SCINet (#67)

* ✨ Add SCINet

* 🍎 Add res connection

README.md

@@ -2,17 +2,27 @@
   <img src="img/tsts-logo.png" width="600"/>
 </div>
 
-[![pypi](https://img.shields.io/pypi/v/tsts?style=flat)](https://pypi.org/project/tsts/0.6.0/)
+[![pypi](https://img.shields.io/pypi/v/tsts?style=flat)](https://pypi.org/project/tsts/0.7.0/)
 [![license](https://img.shields.io/github/license/TakuyaShintate/tsts?style=flat)](https://github.com/TakuyaShintate/tsts/blob/main/LICENSE)
 
 ([Docs](https://takuyashintate.github.io/tsts/))([Benchmark](https://github.com/TakuyaShintate/tsts/tree/main/benchmark/))
 
+## Notice
+
+I am working on benchmarking on the same conditions (now different conditions are used for each model) 🙇‍♂️
+
 ## Introduction
 
 tsts is an open-source easy-to-use toolset for time series forecasting.
 
 ## What's New
 
+### Sep, 21
+
+✨ New Model
+
+SCINet was added. I am working on stacked version of it now.
+
 ### Sep, 10
 
 ✨ New Model  
@@ -65,6 +75,7 @@ Following modules are supported.
         <li>HistricalInertia</li>
         <li>Informer</li>
         <li>NBeats</li>
+        <li>SCINet</li>
         <li>Seq2Seq</li>
       </td>
       <td>

benchmark/informer/informer-ett-h1.yml

@@ -14,7 +14,7 @@ MODEL:
   NUM_H_FEATS: 512
   NUM_ENCODERS: 2
   NUM_DECODERS: 1
-  FF_EXPANSION_RATE: 4.0
+  EXPANSION_RATE: 4.0
   DECODER_IN_LENGTH: 168
   DROPOUT_RATE: 0.05
 

benchmark/scinet/README.md

@@ -0,0 +1,31 @@
+# Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction
+
+## Citation
+
+```
+@article{DBLP:journals/corr/abs-2106-09305,
+  author    = {Minhao Liu and
+               Ailing Zeng and
+               Qiuxia Lai and
+               Qiang Xu},
+  title     = {Time Series is a Special Sequence: Forecasting with Sample Convolution
+               and Interaction},
+  journal   = {CoRR},
+  volume    = {abs/2106.09305},
+  year      = {2021},
+  url       = {https://arxiv.org/abs/2106.09305},
+  eprinttype = {arXiv},
+  eprint    = {2106.09305},
+  timestamp = {Tue, 29 Jun 2021 16:55:04 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2106-09305.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+## ETT
+
+Download the datasets [here](https://github.com/zhouhaoyi/ETDataset/tree/main/ETT-small)
+
+| Type | Input | Target | Lookback | Horizon | MSE | MAE | Ensemble | Script |
+|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| h1 | OT | OT | 96 | 48 | 0.1071 | 0.2559 | No | [train](https://github.com/TakuyaShintate/tsts/tree/main/benchmark/scinet/train_ett_h1.py)/[test](https://github.com/TakuyaShintate/tsts/tree/main/benchmark/scinet/test_ett_h1.py) |

benchmark/scinet/scinet-ett-h1.yml

@@ -0,0 +1,43 @@
+DEVICE: "cuda:0"
+
+IO:
+  LOOKBACK: 96
+  HORIZON: 48
+
+TRAINING:
+  TRAIN_DATA_SPLIT: "col"
+  TRAIN_DATA_RATIO: 0.75
+  NUM_EPOCHS: 100
+
+MODEL:
+  NAME: "SCINet"
+  DEPTH: 3
+  KERNEL_SIZE: 5
+  EXPANSION_RATE: 4.0
+  DROPOUT_RATE: 0.5
+
+DATASET:
+  BASE_START_INDEX: 96
+  NORM_PER_DATASET: False
+
+DATALOADER:
+  BATCH_SIZE_TRAIN: 16
+
+TRAINER:
+  MAX_GRAD_NORM: 1.0
+
+OPTIMIZER:
+  LR: 9.0E-3
+
+LOSSES:
+  NAMES: ["MAE"]
+
+METRICS:
+  NAMES: ["MAE", "MSE"]
+  ARGS: [{}, {}]
+
+SCALER:
+  NAME: "StandardScaler"
+
+LOGGER:
+  LOG_DIR: "scinet-ett-h1"

benchmark/scinet/test_ett_h1.py

@@ -0,0 +1,43 @@
+import pandas as pd
+import torch
+from tqdm import tqdm
+from tsts.metrics import MAE, MSE
+from tsts.scalers import StandardScaler
+from tsts.solvers import TimeSeriesForecaster
+
+lookback = 96
+horizon = 48
+start = 12 * 30 * 24 + 4 * 30 * 24 + lookback
+end = 12 * 30 * 24 + 8 * 30 * 24 + lookback - horizon
+
+X = pd.read_csv("/path/to/ETTh1.csv")
+X = X[["OT"]]
+X = X.values
+X = torch.tensor(X, dtype=torch.float32)
+
+solver = TimeSeriesForecaster("./scinet-ett-h1.yml")
+
+# Initialize scalers with training dataset
+num_train_samples = int(0.75 * (12 * 30 * 24 + 4 * 30 * 24))
+X_scaler = StandardScaler()
+X_scaler.fit(X[:num_train_samples])
+y_scaler = StandardScaler()
+y_scaler.fit(X[:num_train_samples])
+
+metric1 = MSE()
+metric2 = MAE()
+
+for i in tqdm(range(start, end)):
+    x = X[i - lookback : i]
+    y = X[i : i + horizon]
+    y_mask = torch.ones_like(y)
+    x = X_scaler.transform(x)
+    y = y_scaler.transform(y)
+    Z = solver.predict(x)
+    metric1.update(Z, y, y_mask)
+    metric2.update(Z, y, y_mask)
+
+score1 = metric1()
+score2 = metric2()
+print(f"MSE: {score1}")
+print(f"MAE: {score2}")

benchmark/scinet/train_ett_h1.py

@@ -0,0 +1,15 @@
+import pandas as pd
+import torch
+from tsts.solvers import TimeSeriesForecaster
+
+start = 0
+end = 12 * 30 * 24 + 4 * 30 * 24
+
+X = pd.read_csv("/path/to/ETTh1.csv")
+X = X[["OT"]]
+X = X.values
+X = X[start:end]
+X = torch.tensor(X, dtype=torch.float32)
+
+solver = TimeSeriesForecaster("./scinet-ett-h1.yml")
+solver.fit([X])

docs/_sources/index.rst.txt

@@ -20,3 +20,4 @@ Docs
     models/informer
     models/nbeats
     models/seq2seq
+    models/scinet

docs/_sources/models/informer.rst.txt

@@ -22,7 +22,7 @@ Add following lines to config to use *Informer*.
       # Smaller value leads to higher memory efficiency
       CONTRACTION_FACTOR: 5
       # int(NUM_H_FEATS * FF_EXPANSION_RATE) is channel size of conv block after self attention
-      FF_EXPANSION_RATE: 4.0
+      EXPANSION_RATE: 4.0
       # Decoder input series length (last DECODER_IN_LENGTH values are used)
       DECODER_IN_LENGTH: 168
       # Dropout rate

docs/_sources/models/scinet.rst.txt

@@ -0,0 +1,26 @@
+======
+SCINet
+======
+
+How to Use
+==========
+
+Add following lines to config to use *SCINet*.
+
+.. code-block:: yaml
+
+    MODEL:
+      NAME: "SCINet"
+      # Number of levels
+      DEPTH: 3
+      # Kernel size of conv modules
+      KERNEL_SIZE: 5
+      # Expansion rate of conv modules
+      EXPANSION_RATE: 4.0
+      # Dropout rate
+      DROPOUT_RATE: 0.5
+
+Reference
+=========
+
+`Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction <https://arxiv.org/abs/2106.09305?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+arxiv%2FQSXk+%28ExcitingAds%21+cs+updates+on+arXiv.org%29>`_

docs/genindex.html

@@ -128,7 +128,7 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/training.html#step-3-start-training">(Step 3) Start Training</a></li>
 </ul>
 </li>
-<li class="toctree-l1"><a class="reference internal" href="tutorials/inference.html#tutorial-2-inference">(Tutorial 2) Inference</a><ul>
+<li class="toctree-l1"><a class="reference internal" href="tutorials/inference.html">(Tutorial 2) Inference</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#workflow">Workflow</a></li>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#step-1-loading-pre-trained-model">(Step 1) Loading pre-trained model</a></li>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#step-2-initializing-scaler">(Step 2) Initializing scaler</a></li>
@@ -139,7 +139,7 @@ <h2>Table Of Contents</h2>
 </ul>
 <p class="caption" role="heading"><span class="caption-text">Models</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="models/informer.html">Informer</a><ul>
+<li class="toctree-l1"><a class="reference internal" href="models/informer.html#informer">Informer</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="models/informer.html#how-to-use">How to Use</a></li>
 <li class="toctree-l2"><a class="reference internal" href="models/informer.html#reference">Reference</a></li>
 </ul>
@@ -154,6 +154,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="models/seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="models/scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 
@@ -174,7 +179,7 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/training.html#step-3-start-training">(Step 3) Start Training</a></li>
 </ul>
 </li>
-<li class="toctree-l1"><a class="reference internal" href="tutorials/inference.html#tutorial-2-inference">(Tutorial 2) Inference</a><ul>
+<li class="toctree-l1"><a class="reference internal" href="tutorials/inference.html">(Tutorial 2) Inference</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#workflow">Workflow</a></li>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#step-1-loading-pre-trained-model">(Step 1) Loading pre-trained model</a></li>
 <li class="toctree-l2"><a class="reference internal" href="tutorials/inference.html#step-2-initializing-scaler">(Step 2) Initializing scaler</a></li>
@@ -185,7 +190,7 @@ <h2>Table Of Contents</h2>
 </ul>
 <p class="caption" role="heading"><span class="caption-text">Models</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="models/informer.html">Informer</a><ul>
+<li class="toctree-l1"><a class="reference internal" href="models/informer.html#informer">Informer</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="models/informer.html#how-to-use">How to Use</a></li>
 <li class="toctree-l2"><a class="reference internal" href="models/informer.html#reference">Reference</a></li>
 </ul>
@@ -200,6 +205,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="models/seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="models/scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 

docs/index.html

@@ -171,6 +171,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="models/seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="models/scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 
@@ -217,6 +222,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="models/seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="models/scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="models/scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 
@@ -241,6 +251,7 @@ <h1>Docs<a class="headerlink" href="#docs" title="Permalink to this headline">¶
 <li class="toctree-l1"><a class="reference internal" href="models/informer.html">Informer</a></li>
 <li class="toctree-l1"><a class="reference internal" href="models/nbeats.html">NBeats</a></li>
 <li class="toctree-l1"><a class="reference internal" href="models/seq2seq.html">Seq2Seq</a></li>
+<li class="toctree-l1"><a class="reference internal" href="models/scinet.html">SCINet</a></li>
 </ul>
 </div>
 </div>

docs/models/informer.html

@@ -159,7 +159,7 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-1-loading-pre-trained-model">(Step 1) Loading pre-trained model</a></li>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-2-initializing-scaler">(Step 2) Initializing scaler</a></li>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-3-input-data-preparation">(Step 3) Input Data Preparation</a></li>
-<li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-2-running-predict-method">(Step 2) Running <strong>predict</strong> Method</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-4-running-predict-method">(Step 4) Running <strong>predict</strong> Method</a></li>
 </ul>
 </li>
 </ul>
@@ -180,6 +180,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 
@@ -205,7 +210,7 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-1-loading-pre-trained-model">(Step 1) Loading pre-trained model</a></li>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-2-initializing-scaler">(Step 2) Initializing scaler</a></li>
 <li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-3-input-data-preparation">(Step 3) Input Data Preparation</a></li>
-<li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-2-running-predict-method">(Step 2) Running <strong>predict</strong> Method</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../tutorials/inference.html#step-4-running-predict-method">(Step 4) Running <strong>predict</strong> Method</a></li>
 </ul>
 </li>
 </ul>
@@ -226,6 +231,11 @@ <h2>Table Of Contents</h2>
 <li class="toctree-l2"><a class="reference internal" href="seq2seq.html#reference">Reference</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="scinet.html">SCINet</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="scinet.html#how-to-use">How to Use</a></li>
+<li class="toctree-l2"><a class="reference internal" href="scinet.html#reference">Reference</a></li>
+</ul>
+</li>
 </ul>
 
 
@@ -250,7 +260,7 @@ <h2>How to Use<a class="headerlink" href="#how-to-use" title="Permalink to this
   <span class="c1"># Smaller value leads to higher memory efficiency</span>
   <span class="nt">CONTRACTION_FACTOR</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">5</span>
   <span class="c1"># int(NUM_H_FEATS * FF_EXPANSION_RATE) is channel size of conv block after self attention</span>
-  <span class="nt">FF_EXPANSION_RATE</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">4.0</span>
+  <span class="nt">EXPANSION_RATE</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">4.0</span>
   <span class="c1"># Decoder input series length (last DECODER_IN_LENGTH values are used)</span>
   <span class="nt">DECODER_IN_LENGTH</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">168</span>
   <span class="c1"># Dropout rate</span>