Nixtla is an open-source time series forecasting library.

Overview

Nixtla

Nixtla is an open-source time series forecasting library.

We are helping data scientists and developers to have access to open source state-of-the-art forecasting pipelines. For that purpose, we built a complete pipeline that can be deployed in the cloud using AWS and consumed via APIs or consumed as a service. If you want to set up your own infrastructure, follow the instructions in the repository (Azure coming soon).

You can use our fully hosted version as a service through our python SDK (autotimeseries). To consume the APIs on our own infrastructure just request tokens by sending an email to [email protected] or opening a GitHub issue. We currently have free resources available for anyone interested.

We built a fully open-source time-series pipeline capable of achieving 1% of the performance in the M5 competition. Our open-source solution has a 25% better accuracy than Amazon Forecast and is 20% more accurate than fbprophet. It also performs 4x faster than Amazon Forecast and is less expensive.

To reproduce the results: Open In Colab or you can read this Medium Post.

At Nixtla we strongly believe in open-source, so we have released all the necessary code to set up your own time-series processing service in the cloud (using AWS, Azure is WIP). This repository uses continuous integration and deployment to deploy the APIs on our infrastructure.

Python SDK Basic Usage

CI python sdk

Install

PyPI

pip install autotimeseries

How to use

Check the following examples for a full pipeline:

Basic usage

import os

from autotimeseries.core import AutoTS

autotimeseries = AutoTS(bucket_name=os.environ['BUCKET_NAME'],
                        api_id=os.environ['API_ID'],
                        api_key=os.environ['API_KEY'],
                        aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
                        aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'])

Upload dataset to S3

train_dir = '../data/m5/parquet/train'
# File with target variables
filename_target = autotimeseries.upload_to_s3(f'{train_dir}/target.parquet')
# File with static variables
filename_static = autotimeseries.upload_to_s3(f'{train_dir}/static.parquet')
# File with temporal variables
filename_temporal = autotimeseries.upload_to_s3(f'{train_dir}/temporal.parquet')

Each time series of the uploaded datasets is defined by the column item_id. Meanwhile the time column is defined by timestamp and the target column by demand. We need to pass this arguments to each call.

columns = dict(unique_id_column='item_id',
               ds_column='timestamp',
               y_column='demand')

Send the job to make forecasts

response_forecast = autotimeseries.tsforecast(filename_target=filename_target,
                                              freq='D',
                                              horizon=28,
                                              filename_static=filename_static,
                                              filename_temporal=filename_temporal,
                                              objective='tweedie',
                                              metric='rmse',
                                              n_estimators=170,
                                              **columns)

Download forecasts

autotimeseries.download_from_s3(filename='forecasts_2021-10-12_19-04-32.csv', filename_output='../data/forecasts.csv')

Forecasting Pipeline as a Service

Our forecasting pipeline is modular and built upon simple APIs:

tspreprocess

CI/CD tspreprocess Lambda CI/CD tspreprocess docker image

Time series usually contain missing values. This is the case for sales data where only the events that happened are recorded. In these cases it is convenient to balance the panel, i.e., to include the missing values to correctly determine the value of future sales.

The tspreprocess API allows you to do this quickly and easily. In addition, it allows one-hot encoding of static variables (specific to each time series, such as the product family in case of sales) automatically.

tsfeatures

CI/CD tsfeatures Lambda CI/CD tsfeatures docker image

It is usually good practice to create features of the target variable so that they can be consumed by machine learning models. This API allows users to create features at the time series level (or static features) and also at the temporal level.

The tsfeatures API is based on the tsfeatures library also developed by the Nixtla team (inspired by the R package tsfeatures) and the tsfresh library.

With this API the user can also generate holiday variables. Just enter the country of the special dates or a file with the specific dates and the API will return dummy variables of those dates for each observation in the dataset.

tsforecast

CI/CD tsforecast Lambda CI/CD tsforecast docker image

The tsforecast API is responsible for generating the time series forecasts. It receives as input the target data and can also receive static variables and time variables. At the moment, the API uses the mlforecast library developed by the Nixtla team using LightGBM as a model.

In future iterations, the user will be able to choose different Deep Learning models based on the nixtlats library developed by the Nixtla team.

tsbenchmarks

CI/CD tsbenchmarks Lambda CI/CD tsbenchmarks docker image

The tsbenchmarks API is designed to easily compare the performance of models based on time series competition datasets. In particular, the API offers the possibility to evaluate forecasts of any frequency of the M4 competition and also of the M5 competition.

These APIs, written in Python and can be consumed through an SDK also written in Python. The following diagram summarizes the structure of our pipeline:

Build your own time-series processing service using AWS

Why ?

We want to contribute to open source and help data scientists and developers to achieve great forecasting results without the need to implement complex pipelines.

How?

If you want to use our hosted version send us an email or open a github issue and ask for API Keys.

If you want to deploy Nixtla on your own AWS Cloud you will need:

  • API Gateway (to handle API calls).
  • Lambda (or some computational unit).
  • SageMaker (or some bigger computational unit).
  • ECR (to store Docker images).
  • S3 (for inputs and outputs).

You will end with an architecture that looks like the following diagram

Each call to the API executes a particular Lambda function depending on the endpoint. That particular lambda function instantiates a SageMaker job using a predefined type of instance. Finally, SageMaker reads the input data from S3 and writes the processed data to S3, using a predefined Docker image stored in ECR.

Run the API locally

  1. Create the environment using make init.
  2. Launch the app using make app.

Create AWS resources

Create S3 buckets

For each service:

  1. Create an S3 bucket. The code of each lambda function will be uploaded here.

Create ECR repositorires

For each service:

  1. Create a private repository for each service.

Lambda Function

For each service:

  1. Create a lambda function with Python 3.7 runtime.
  2. Modify the runtime setting and enter main.handler in the handler.
  3. Go to the configuration:
    • Edit the general configuration and add a timeout of 9:59.
    • Add an existing role capable of reading/writing from/to S3 and running Sagemaker services.
  4. Add the following environment variables:
    • PROCESSING_REPOSITORY_URI: ECR URI of the docker image corresponding to the service.
    • ROLE: A role capable of reading/writing from/to S3 and also running Sagemaker services.
    • INSTANCE_COUNT
    • INSTANCE_TYPE

API Gateway

  1. Create a public REST API (Regional).
  2. For each endpoint in api/main.py… add a resource.
  3. For each created method add an ANY method:
    • Select lambda function.
    • Select Use Lambda Proxy Integration.
    • Introduce the name of the lambda function linked to that resource.
    • Once the method is created select Method Request and set API key required to true.
  4. Deploy the API.

Usage plan

  1. Create a usage plan based on your needs.
  2. Add your API stage.

API Keys

  1. Generate API keys as needed.

Deployment

GitHub secrets

  1. Set the following secrets in your repo:
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_DEFAULT_REGION
Owner
Nixtla
Open Source Time Series Forecasting
Nixtla
Client - 🔥 A tool for visualizing and tracking your machine learning experiments

Weights and Biases Use W&B to build better models faster. Track and visualize all the pieces of your machine learning pipeline, from datasets to produ

Weights & Biases 5.2k Jan 03, 2023
A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

AI Fairness 360 (AIF360) The AI Fairness 360 toolkit is an extensible open-source library containg techniques developed by the research community to h

1.9k Jan 06, 2023
PyNNDescent is a Python nearest neighbor descent for approximate nearest neighbors.

PyNNDescent PyNNDescent is a Python nearest neighbor descent for approximate nearest neighbors. It provides a python implementation of Nearest Neighbo

Leland McInnes 699 Jan 09, 2023
Iris species predictor app is used to classify iris species created using python's scikit-learn, fastapi, numpy and joblib packages.

Iris Species Predictor Iris species predictor app is used to classify iris species using their sepal length, sepal width, petal length and petal width

Siva Prakash 5 Apr 05, 2022
Using Logistic Regression and classifiers of the dataset to produce an accurate recall, f-1 and precision score

Using Logistic Regression and classifiers of the dataset to produce an accurate recall, f-1 and precision score

Thines Kumar 1 Jan 31, 2022
Model Agnostic Confidence Estimator (MACEST) - A Python library for calibrating Machine Learning models' confidence scores

Model Agnostic Confidence Estimator (MACEST) - A Python library for calibrating Machine Learning models' confidence scores

Oracle 95 Dec 28, 2022
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. 10x Larger Models 10x Faster Trainin

Microsoft 8.4k Dec 30, 2022
Nevergrad - A gradient-free optimization platform

Nevergrad - A gradient-free optimization platform nevergrad is a Python 3.6+ library. It can be installed with: pip install nevergrad More installati

Meta Research 3.4k Jan 08, 2023
虚拟货币(BTC、ETH)炒币量化系统项目。在一版本的基础上加入了趋势判断

🎉 第二版本 🎉 (现货趋势网格) 介绍 在第一版本的基础上 趋势判断,不在固定点位开单,选择更优的开仓点位 优势: 🎉 简单易上手 安全(不用将api_secret告诉他人) 如何启动 修改app目录下的authorization文件

幸福村的码农 250 Jan 07, 2023
Machine Learning Algorithms ( Desion Tree, XG Boost, Random Forest )

implementation of machine learning Algorithms such as decision tree and random forest and xgboost on darasets then compare results for each and implement ant colony and genetic algorithms on tsp map,

Mohamadreza Rezaei 1 Jan 19, 2022
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Jan 02, 2023
A simple guide to MLOps through ZenML and its various integrations.

ZenBytes Join our Slack Community and become part of the ZenML family Give the main ZenML repo a GitHub star to show your love ZenBytes is a series of

ZenML 127 Dec 27, 2022
A Python library for choreographing your machine learning research.

A Python library for choreographing your machine learning research.

AI2 270 Jan 06, 2023
Tribuo - A Java machine learning library

Tribuo - A Java prediction library (v4.1) Tribuo is a machine learning library in Java that provides multi-class classification, regression, clusterin

Oracle 1.1k Dec 28, 2022
AutoTabular automates machine learning tasks enabling you to easily achieve strong predictive performance in your applications.

AutoTabular AutoTabular automates machine learning tasks enabling you to easily achieve strong predictive performance in your applications. With just

wenqi 2 Jun 26, 2022
A Collection of Conference & School Notes in Machine Learning 🦄📝🎉

Machine Learning Conference & Summer School Notes. 🦄📝🎉

558 Dec 28, 2022
Price forecasting of SGB and IRFC Bonds and comparing there returns

Project_Bonds Project Title : Price forecasting of SGB and IRFC Bonds and comparing there returns. Introduction of the Project The 2008-09 global fina

Tishya S 1 Oct 28, 2021
A collection of interactive machine-learning experiments: 🏋️models training + 🎨models demo

🤖 Interactive Machine Learning experiments: 🏋️models training + 🎨models demo

Oleksii Trekhleb 1.4k Jan 06, 2023
Markov bot - A Writing bot based on Markov Chain for Data Structure Lab

基于马尔可夫链的写作机器人 前端 用html/css完成 Demo展示(已给出文本的相应展示) 用户提供相关的语料库后训练的成果 后端 要完成的几个接口 解析文

DysprosiumDy 9 May 05, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23.6k Jan 03, 2023