LONG-TERM SERIES FORECASTING WITH QUERYSELECTOR – EFFICIENT MODEL OF SPARSEATTENTION

Overview

Query Selector

Here you can find code and data loaders for the paper https://arxiv.org/pdf/2107.08687v1.pdf . Query Selector is a novel approach to sparse attention Transformer algorithm that is especially suitable for long term time series forecasting

Depencency

Python            3.7.9
deepspeed         0.4.0
numpy             1.20.3
pandas            1.2.4
scipy             1.6.3
tensorboardX      1.8
torch             1.7.1
torchaudio        0.7.2
torchvision       0.8.2
tqdm              4.61.0

Results on ETT dataset

Univariate

Data Prediction len Informer MSE Informer MAE Trans former MSE Trans former MAE Query Selector MSE Query Selector MAE MSE ratio
ETTh1 24 0.0980 0.2470 0.0548 0.1830 0.0436 0.1616 0.445
ETTh1 48 0.1580 0.3190 0.0740 0.2144 0.0721 0.2118 0.456
ETTh1 168 0.1830 0.3460 0.1049 0.2539 0.0935 0.2371 0.511
ETTh1 336 0.2220 0.3870 0.1541 0.3201 0.1267 0.2844 0.571
ETTh1 720 0.2690 0.4350 0.2501 0.4213 0.2136 0.3730 0.794
ETTh2 24 0.0930 0.2400 0.0999 0.2479 0.0843 0.2239 0.906
ETTh2 48 0.1550 0.3140 0.1218 0.2763 0.1117 0.2622 0.721
ETTh2 168 0.2320 0.3890 0.1974 0.3547 0.1753 0.3322 0.756
ETTh2 336 0.2630 0.4170 0.2191 0.3805 0.2088 0.3710 0.794
ETTh2 720 0.2770 0.4310 0.2853 0.4340 0.2585 0.4130 0.933
ETTm1 24 0.0300 0.1370 0.0143 0.0894 0.0139 0.0870 0.463
ETTm1 48 0.0690 0.2030 0.0328 0.1388 0.0342 0.1408 0.475
ETTm1 96 0.1940 0.2030 0.0695 0.2085 0.0702 0.2100 0.358
ETTm1 288 0.4010 0.5540 0.1316 0.2948 0.1548 0.3240 0.328
ETTm1 672 0.5120 0.6440 0.1728 0.3437 0.1735 0.3427 0.338

Multivariate

Data Prediction len Informer MSE Informer MAE Trans former MSE Trans former MAE Query Selector MSE Query Selector MAE MSE ratio
ETTh1 24 0.5770 0.5490 0.4496 0.4788 0.4226 0.4627 0.732
ETTh1 48 0.6850 0.6250 0.4668 0.4968 0.4581 0.4878 0.669
ETTh1 168 0.9310 0.7520 0.7146 0.6325 0.6835 0.6088 0.734
ETTh1 336 1.1280 0.8730 0.8321 0.7041 0.8503 0.7039 0.738
ETTh1 720 1.2150 0.8960 1.1080 0.8399 1.1150 0.8428 0.912
ETTh2 24 0.7200 0.6650 0.4237 0.5013 0.4124 0.4864 0.573
ETTh2 48 1.4570 1.0010 1.5220 0.9488 1.4074 0.9317 0.966
ETTh2 168 3.4890 1.5150 1.6225 0.9726 1.7385 1.0125 0.465
ETTh2 336 2.7230 1.3400 2.6617 1.2189 2.3168 1.1859 0.851
ETTh2 720 3.4670 1.4730 3.1805 1.3668 3.0664 1.3084 0.884
ETTm1 24 0.3230 0.3690 0.3150 0.3886 0.3351 0.3875 0.975
ETTm1 48 0.4940 0.5030 0.4454 0.4620 0.4726 0.4702 0.902
ETTm1 96 0.6780 0.6140 0.4641 0.4823 0.4543 0.4831 0.670
ETTm1 288 1.0560 0.7860 0.6814 0.6312 0.6185 0.5991 0.586
ETTm1 672 1.1920 0.9260 1.1365 0.8572 1.1273 0.8412 0.946

State Of Art

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

Citation

@misc{klimek2021longterm,
      title={Long-term series forecasting with Query Selector -- efficient model of sparse attention}, 
      author={Jacek Klimek and Jakub Klimek and Witold Kraskiewicz and Mateusz Topolewski},
      year={2021},
      eprint={2107.08687},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Contact

If you have any questions please contact us by email - [email protected]

Owner
MORAI
MORAI
Your interactive network visualizing dashboard

Your interactive network visualizing dashboard Documentation: Here What is Jaal Jaal is a python based interactive network visualizing tool built usin

Mohit 177 Jan 04, 2023
O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning (CoRL 2021)

O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning Object-object Interaction Affordance Learning. For a given object-object int

Kaichun Mo 26 Nov 04, 2022
Library for 8-bit optimizers and quantization routines.

bitsandbytes Bitsandbytes is a lightweight wrapper around CUDA custom functions, in particular 8-bit optimizers and quantization functions. Paper -- V

Facebook Research 687 Jan 04, 2023
The official code repository for examples in the O'Reilly book 'Generative Deep Learning'

Generative Deep Learning Teaching Machines to paint, write, compose and play The official code repository for examples in the O'Reilly book 'Generativ

David Foster 1.3k Dec 29, 2022
StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation

StyleGAN2 with adaptive discriminator augmentation (ADA) — Official TensorFlow implementation Training Generative Adversarial Networks with Limited Da

NVIDIA Research Projects 1.7k Dec 29, 2022
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 0 Feb 06, 2022
This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Diverse Motion Stylization (Official) This is the official Pytorch implementation of this paper. Diverse Motion Stylization for Multiple Style Domains

Soomin Park 28 Dec 16, 2022
Miscellaneous and lightweight network tools

Network Tools Collection of miscellaneous and lightweight network tools to simplify daily operations, administration, and troubleshooting of networks.

Nicholas Russo 22 Mar 22, 2022
Codes for NeurIPS 2021 paper "Adversarial Neuron Pruning Purifies Backdoored Deep Models"

Adversarial Neuron Pruning Purifies Backdoored Deep Models Code for NeurIPS 2021 "Adversarial Neuron Pruning Purifies Backdoored Deep Models" by Dongx

Dongxian Wu 31 Dec 11, 2022
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
PyTorch implementation of Spiking Neural Networks trained on surrogate gradient & BPTT using snntorch.

snn-localization repo PyTorch implementation of Spiking Neural Networks trained on surrogate gradient & BPTT using snntorch. Install Dependencies Orig

Sami BARCHID 1 Jan 06, 2022
Unofficial PyTorch Implementation of Multi-Singer

Multi-Singer Unofficial PyTorch Implementation of Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus. Requirements See re

SunMail-hub 123 Dec 28, 2022
A trusty face recognition research platform developed by Tencent Youtu Lab

Introduction TFace: A trusty face recognition research platform developed by Tencent Youtu Lab. It provides a high-performance distributed training fr

Tencent 956 Jan 01, 2023
Investigating Attention Mechanism in 3D Point Cloud Object Detection (arXiv 2021)

Investigating Attention Mechanism in 3D Point Cloud Object Detection (arXiv 2021) This repository is for the following paper: "Investigating Attention

52 Nov 19, 2022
Code for the Paper: Conditional Variational Capsule Network for Open Set Recognition

Conditional Variational Capsule Network for Open Set Recognition This repository hosts the official code related to "Conditional Variational Capsule N

Guglielmo Camporese 35 Nov 21, 2022
Code repository for the paper "Tracking People with 3D Representations"

Tracking People with 3D Representations Code repository for the paper "Tracking People with 3D Representations" (paper link) (project site). Jathushan

Jathushan Rajasegaran 77 Dec 03, 2022
Image Segmentation Evaluation

Image Segmentation Evaluation Martin Keršner, [email protected] Evaluation

Martin Kersner 273 Oct 28, 2022
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

CoTr: Efficient 3D Medical Image Segmentation by bridging CNN and Transformer This is the official pytorch implementation of the CoTr: Paper: CoTr: Ef

218 Dec 25, 2022
NeuralCompression is a Python repository dedicated to research of neural networks that compress data

NeuralCompression is a Python repository dedicated to research of neural networks that compress data. The repository includes tools such as JAX-based entropy coders, image compression models, video c

Facebook Research 297 Jan 06, 2023
Code for the TIP 2021 Paper "Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss"

PurNet Project for the TIP 2021 Paper "Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss" Abstract Image-based salie

Jinming Su 4 Aug 25, 2022