AutoTimes (Large Language Models for Time Series Forecasting)

Official implementation: AutoTimes: Autoregressive Time Series Forecasters via Large Language Models. [Slides], [Poster]

Time Series Forecasting: AutoTimes convert LLMs to autoregressive time series forecasters. Unlike previous methods, the forecaster can accommodate arbitrary-length lookback and predictions.

Zero-Shot Forecasting: AutoTimes takes advantage of LLM's general-purposed token transition as the future extrapolation of time series, demonstrating good performance without downstream samples.

In-Context Forecasting: We propose in-context forecasting for the first time, where time series prompts can be incorporated into the input context to enhance forecasting.

Easy-to-Use: AutoTimes is compatiable with any decoder-only large language models, demonstrating generality and proper scaling behavior.

Updates

🚩 News (2024.10): An introduction of our works in available [Slides]. See you at NeurIPS 2024!

🚩 News (2024.10): AutoTimes has been accepted by NeurIPS 2024. A revised version (25 Pages) is now available, including prompt engineering of in-context forecasting, adaptation cost evaluations, textual embeddings of metadata, and low-rank adaptation techique.

🚩 News (2024.08): Recent work (code) has also raised questions about previous non-autoregressive LLM4TS methods. We conduct ablations here, highlighting AutoTimes can truly utilize LLMs. Instead of adopting LLMs in a BERT-style, the general-purpose token transition is transferable among time series and natural language.

🚩 News (2024.2) Scripts for the above tasks in our paper are all available.

Introduction

🌟 While prevalent LLM4TS methods adapt LLMs as encoder-only and non-autoregressive forecasters, we propose to keep consistent with the inherent autoregressive property and model architecture.

💪 We aim to fully revitalize LLMs as foundation models for time series forecasting, including multi-step forecasting, zero-shot capability, in-context forecasting, and multimodal utilization.

🏆 AutoTimes achieves state-of-the-art performance with 0.1% trainable parameters and over 5× training/inference speedup compared to advanced LLM-based forecasters.

Usage

1. Install Pytorch and necessary dependencies.

pip install -r requirements.txt

1. Put the datasets [Google Drive] [Tsinghua Cloud] under the folder ./dataset/.

2. Download the large language models from Hugging Face. The default LLM is LLaMA-7B, you can change the llm_ckp_dir in run.py to use other LLMs.

   • LLaMA-7B
   • OPT Family
   • GPT2

   For example, if you download and put the LLaMA directory successfully, the directory structure is as follows:

   • data_provider
   • dataset
   • llama
     • config.json
     • pytorch_model-00001-of-00002.bin
     • pytorch_model-00002-of-00002.bin
     • ...
   • ...
   • run.py

3. Using the position embedding from textual timestamps. Note that we have provided the embeddings of the given datasets in the download links, which are generated by LLaMA, suffixed by {dataset_name}.pt. If you want to generate the embeddings from your customized datasets, please refer to the following codes:

# preprocess timestamps to generate text embedding
python ./preprocess.py --gpu 0 --dataset ETTh1

4. Train and evaluate the model. We provide all the above tasks under the folder ./scripts/.

# the default large language model is LLaMA-7B

# long-term forecasting
bash ./scripts/time_series_forecasting/long_term/AutoTimes_ETTh1.sh

# short-term forecasting
bash ./scripts/time_series_forecasting/short_term/AutoTimes_M4.sh

# zero-shot forecasting
# it's worth noting that sM4_tM3 utilizes models trained
# on short-term, you should run AutoTimes_M4 first
bash ./scripts/zero_shot_forecasting/sM4_tM3.sh
bash ./scripts/zero_shot_forecasting/sM3_tM4.sh

# in-context forecasting
bash ./scripts/in_context_forecasting/M3.sh

# try on other large language models
bash ./scripts/method_generality/opt.sh

Due to the simple tokenization and the frozen of LLM blocks, AutoTimes is highly compatiable with LLMs. For example, it requires only 15min for AutoTime to repurpuse LLaMA-7B on ETTh1 on one single RTX 3090-24G.

A Usage Example

See predict.ipynb for a simple training and inference workflow.

Overall Approach

• Time series and corresponding timestamps are segmented.
• Textual timestamps are converted into the position embedding of segments.
• AutoTimes learns to embed time series segments by next token prediction, where intermediate layers of LLM are frozen.

Capability

	Non-Autoregressive	Autoregressive
Training	Trained with specific lookback-forecast lengths	Trained with the context length with each generated token being supervised
One-step Forecasting	Applicable only on fixed lookback-forecast lengths	Flexible on scenarios less than the context length like large language models
Rolling Forecasting	Has to drop the lookback series because of the fixed input length	Can prolong the lookback horizon until the total length exceeds the context length

Time Series Forecasting

Towards the versatility of foundation models, we establish a novel one-for-all benchmark: a single forecaster is trained on one dataset and subsequently utilized for all prediction lengths.

Zero-Shot Forecasting

We evaluate the performance under the transfer learning scenario, where the forecaster is first trained on a source domain and then evaluated on the unseen target domain.

In-Context Forecasting

We concatenate time series prompts with lookback series and feed them as the context of the forecaster, termed in-context forecasting.

Benefiting from time series prompts from the target domain, AUTOTIMES achieves consistent promotions (averaged 13.3% SMAPE reduction) compared with zero-shot forecasting.

Model Generality

We evaluate the generality and efficiency on other LLMs, demonstrating improved performance with the increase of parameters that validates the scaling law.

Method Efficiency

Not only does AutoTime achieve more acurate predcitions but its training and reasoning time is also greatly reduced, bringing over 5× speedup on average.

Showcases

We investigate different prompt retrieval strategies. Insightful results are provided to reveal the influence of using time series prompts for interactive prediction.

Citation

If you find this repo helpful, please cite our paper.

@article{liu2024autotimes,
  title={AutoTimes: Autoregressive Time Series Forecasters via Large Language Models},
  author={Liu, Yong and Qin, Guo and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  journal={arXiv preprint arXiv:2402.02370},
  year={2024}
}

Acknowledgement

We appreciate the following GitHub repos a lot for their valuable code and efforts.

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)
• FPT (https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All)

Contact

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Guo Qin (qinguo24@mails.tsinghua.edu.cn)