A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.

Overview

Xcessiv

PyPI license PyPI Build Status

Xcessiv is a tool to help you create the biggest, craziest, and most excessive stacked ensembles you can think of.

Stacked ensembles are simple in theory. You combine the predictions of smaller models and feed those into another model. However, in practice, implementing them can be a major headache.

Xcessiv holds your hand through all the implementation details of creating and optimizing stacked ensembles so you're free to fully define only the things you care about.

The Xcessiv process

Define your base learners and performance metrics

define_base_learner

Keep track of hundreds of different model-hyperparameter combinations

list_base_learner

Effortlessly choose your base learners and create an ensemble with the click of a button

ensemble

Features

  • Fully define your data source, cross-validation process, relevant metrics, and base learners with Python code
  • Any model following the Scikit-learn API can be used as a base learner
  • Task queue based architecture lets you take full advantage of multiple cores and embarrassingly parallel hyperparameter searches
  • Direct integration with TPOT for automated pipeline construction
  • Automated hyperparameter search through Bayesian optimization
  • Easy management and comparison of hundreds of different model-hyperparameter combinations
  • Automatic saving of generated secondary meta-features
  • Stacked ensemble creation in a few clicks
  • Automated ensemble construction through greedy forward model selection
  • Export your stacked ensemble as a standalone Python file to support multiple levels of stacking

Installation and Documentation

You can find installation instructions and detailed documentation hosted here.

FAQ

Where does Xcessiv fit in the machine learning process?

Xcessiv fits in the model building part of the process after data preparation and feature engineering. At this point, there is no universally acknowledged way of determining which algorithm will work best for a particular dataset (see No Free Lunch Theorem), and while heuristic optimization methods do exist, things often break down into trial and error as you try to find the best model-hyperparameter combinations.

Stacking is an almost surefire method to improve performance beyond that of any single model, however, the complexity of proper implementation often makes it impractical to apply them in practice outside of Kaggle competitions. Xcessiv aims to make the construction of stacked ensembles as painless as possible and lower the barrier for entry.

I don't care about fancy stacked ensembles and what not, should I still use Xcessiv?

Absolutely! Even without the ensembling functionality, the sheer amount of utility provided by keeping track of the performance of hundreds, and even thousands of ML models and hyperparameter combinations is a huge boon.

How does Xcessiv generate meta-features for stacking?

You can choose whether to generate meta-features through cross-validation (stacked generalization) or with a holdout set (blending). You can read about these two methods and a lot more about stacked ensembles in the Kaggle Ensembling Guide. It's a great article and provides most of the inspiration for this project.

Contributing

Xcessiv is in its very early stages and needs the open-source community to guide it along.

There are many ways to contribute to Xcessiv. You could report a bug, suggest a feature, submit a pull request, improve documentation, and many more.

If you would like to contribute something, please visit our Contributor Guidelines.

Project Status

Xcessiv is currently in alpha and is unstable. Future versions are not guaranteed to be backwards-compatible with current project files.

Comments
  • Can't Use

    Can't Use

    Sorry for what is no doubt a stupid question:

    I've started Redis via redis-server. It says it's running on port 6379. Then I run xcessiv, but it takes me to a page that's not found. The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.. Any idea what I can do? I'm really eager to use Xcessiv.

    opened by xnmp 6
  • Automated ensembling techniques

    Automated ensembling techniques

    Working for a while with Xcessiv, I feel there's a need for some way to automate the selection of base learners in an ensemble. I'm unaware of existing techniques for this, so if anyone has any suggestions or could point me towards relevant literature, it would be greatly appreciated.

    enhancement 
    opened by reiinakano 5
  • Added more of the sklearn regressors to the presets

    Added more of the sklearn regressors to the presets

    Added the large majority of the more popular regressors of sklearn. I am aware that a few may be missing. Also, I tidied the code slightly and split the regressors and classifiers into two sections.

    opened by enisnazif 4
  • Memory management

    Memory management

    First of all, thanks! I find this project fascinating. My question/issue is about how do you handle the memory for multiple processes. By default Python will create a copy of the data per process. This is prohibitive for large datasets.

    How did you manage this problem?

    opened by alvarouc 3
  • Move .gitignore to project root and add Python ignores

    Move .gitignore to project root and add Python ignores

    I think the best practice for .gitignore is to have a single .gitignore file at the root of the project so I moved the .gitignore that was in xcessiv/ui (I think it was generated by create-react-scripts) to the project root and added some Python ignore lines.

    opened by menglewis 3
  • Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Cross validation as part of #15

    I'm keen to finish implementing all of the cv / metrics within sklearn, just wanted to make sure I was doing it right since this is my first pull request!

    opened by enisnazif 2
  • XGBRegressor model stuck in queued status

    XGBRegressor model stuck in queued status

    I tried to make a regression model to run on zillow data from kaggle available here https://www.kaggle.com/c/zillow-prize-1/data Here is a gist of my dataset extractor as well as setting up the XGBRegressor and an exception that was the last thing left in the console https://gist.github.com/jef5ez/a9b0650293f343682a58b0f0500f3332 I selected the shuffle split for both cross validation settings and added MSE as the learner metric. The base learner seems to verify fine on the boston housing data. After hitting finalize and selecting a single base learner a row shows up below but is stuck in the Queued status.

    python 3.5.2 xcessiv (0.2.2) xgboost (0.6a2)

    opened by jef5ez 2
  • The _BasePipeline in exported Python script should be _BaseComposition

    The _BasePipeline in exported Python script should be _BaseComposition

    Since scitkit-learn 0.19.x, the base class for Pipeline has changed to _BaseComposition. https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py When using the generated code for training, it raises a name-not-found error on newer versions of sklearn. At the moment, an easy workaround is to change two instances of the word manually in the generated script.

    opened by Mithrillion 1
  • Issues with TfidnVectorizer

    Issues with TfidnVectorizer

    Hey, great tool.

    I have a problem though when I am trying to use a TfidfVectorizer for Text Classification. When I create a Single Base Learner I get the error:

    ValueError: all the input array dimensions except for the concatenation axis must match exactly .

    The type of the X variable is an numpy.ndarray, but if I don't convert the variable X to an array then I get the error message:

    TypeError: Singleton array array(<92820x194 sparse matrix of type '<class 'numpy.float64'>' with 92820 stored elements in Compressed Sparse Row format>, dtype=object) cannot be considered a valid collection.

    I choose the preset learner setting scikit-learn Random Forest as a Base Learner Type.

    import os
    import numpy as np
    import pandas as pd
    import pickle
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.ensemble import RandomForestClassifier
    
    def extract_main_dataset():
        # pandas data frame with the columns Classification, FeatureVector
        # ie:
        # 0, 'This is the feature vector'
        # 1, 'This is another feature vector' 
        # 2, 'This is yet another feature vector' 
        # 1, 'This is the last feature vector example' 
        with open('feature_vector.pik', 'rb') as rf:
            feature_vector = pickle.load(rf)
    
        y = np.array(feature_vector.Classification.values)
        title_rf_vectorizer = TfidfVectorizer(ngram_range=(2, 9),
                                              sublinear_tf=True,
                                              use_idf=True,
                                              strip_accents='ascii')
    
        title_rf_classifier = RandomForestClassifier(n_estimators=100, n_jobs=8)
        X = title_rf_vectorizer.fit_transform(feature_vector["Classification"]).toarray()
        return X, y
    
    opened by bbowler86 1
  • Valid values for metric to optimise in bayesian optimisation?

    Valid values for metric to optimise in bayesian optimisation?

    Is there a list of valid metric_to_optimise for Bayesian Optimisation?

    I am using sklearn mean_squared_regression for my base learning but when I enter that into the Bayesian Optimisation menu under metric_to_optimise I get:

    assert module.metric_to_optimize in automated_run.base_learner_origin.metric_generators
    AssertionError
    
    question 
    opened by Data-drone 1
  • 'dict_keys' object does not support indexing

    'dict_keys' object does not support indexing

    On lines 306 and 309 of views.py, trying to index a dictionary keys object will fail on Python 3 and result in a server error. The fix is simple: change all occurrences of

    base_learner_origin.validation_results.keys()[0]

    to

    list(base_learner_origin.validation_results.keys())[0]

    opened by KhaledSharif 1
  • redis.exceptions.DataError at xcessiv launch

    redis.exceptions.DataError at xcessiv launch

    Hello, When I try to launch xcessiv I get an error:

    Traceback (most recent call last): File "/PATH_TO/anaconda3/bin/xcessiv", line 10, in <module> sys.exit(main()) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/xcessiv/scripts/runapp.py", line 51, in main redis_conn.get(None) # will throw exception if Redis is unavailable File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 1264, in get return self.execute_command('GET', name) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 774, in execute_command connection.send_command(*args) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 620, in send_command self.send_packed_command(self.pack_command(*args)) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 663, in pack_command for arg in imap(self.encoder.encode, args): File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 125, in encode "byte, string or number first." % typename) redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a byte, string or number first.

    Previously I had to change from gevent.wsgi import WSGIServer to from gevent.pywsgi import WSGIServer as indicated in this issue

    My server is responding when I do redis-cli ping

    I am on Ubuntu 18.04, with python 3.7.3 and redis 5.0.5

    Do you have an idea to fix this? Thanks!

    opened by AlexCoul 0
  • How to import homemade modules in Xcessiv?

    How to import homemade modules in Xcessiv?

    I'm trying to import homemade module named preprocessing_115v (filename preprocessing_115v.py) into the main data extraction source code but I can't seem to find it :

    ############# import preprocessing_115v <-- where do I store the preprocessing_115v.py file for it to load here? def extract_main_dataset(): import pandas as pd df=pd.read_csv('./data.csv', sep=',',header=None) X=df.values labels=pd.read_csv('./labelsnum.csv', sep=',',header=None) y=labels.values y=y[:,0] return X, y ##############

    Amazing program by the way :-)

    opened by fcoppey 0
  • xcessiv server

    xcessiv server

    Hi This project looks very cool, but I am having some problems with the setup. I am running this in a container (my own), and I can't get the server to show up. From inside the container I can see the server running - ps shows xcessiv running and curl localhost:1994 gives me some HTML from xcessiv. From outside the container, however, there's nothing.

    I suppose that's down to the server.py file which I have now changed to this:

      1 from __future__ import absolute_import, print_function, division, unicode_literals¬                                                                              
      2 from gevent.wsgi import WSGIServer¬
      3 # import webbrowser¬
      4 ¬
      5 ¬
      6 def launch(app):¬
      7     http_server = WSGIServer(('0.0.0.0', app.config['XCESSIV_PORT']), app)¬
      8     # webbrowser.open_new('http://localhost:' + str(app.config['XCESSIV_PORT']))¬
      9     http_server.serve_forever()¬
    

    I have changed the WSGIServer setup to be open to outside connection (I suppose that's what I changed), but it's still not showing up.

    Feedback appreciated. I'd like to try this out. Thanks!

    opened by benman1 0
  • Feature Request - Backup .db file

    Feature Request - Backup .db file

    I got an error something to the effect of "Error with JSON "N" at position 8345", presumably caused by my manually editing the code for one of the base learners. Once I got this error however, none of the base learners in my project would load. I resolved it by manually deleting the base learner I had been editing from the .db file. I'll post the specifics if I can recreate it, but I'm wondering if it might be prudent to have some kind of db backup/"Last Known Good Configuration"?

    opened by Tahlor 0
  • Fix issue #63 no module named wsgi

    Fix issue #63 no module named wsgi

    In file server.py

    from gevent.wsgi import WSGIServer
    

    Has to be changed to:

    from gevent.pywsgi import WSGIServer
    

    http://www.gevent.org/api/gevent.pywsgi.html

    opened by KhaledTo 0
  • ImportError: No module named wsgi

    ImportError: No module named wsgi

    File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xcessiv/server.py", line 2, in from gevent.wsgi import WSGIServer ImportError: No module named wsgi

    opened by xialeizhou 2
Releases(v0.5.1)
Owner
Reiichiro Nakano
I like working on awesome things with awesome people!
Reiichiro Nakano
Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022)

Blockwise Sequential Model Learning Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022) For ins

2 Jun 17, 2022
TreeSubstitutionCipher - Encryption system based on trees and substitution

Tree Substitution Cipher Generation Algorithm: Generate random tree. Tree nodes

stepa 1 Jan 08, 2022
Code for LIGA-Stereo Detector, ICCV'21

LIGA-Stereo Introduction This is the official implementation of the paper LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based

Xiaoyang Guo 75 Dec 09, 2022
Satellite labelling tool for manual labelling of storm top features such as overshooting tops, above-anvil plumes, cold U/Vs, rings etc.

Satellite labelling tool About this app A tool for manual labelling of storm top features such as overshooting tops, above-anvil plumes, cold U/Vs, ri

Czech Hydrometeorological Institute - Satellite Department 10 Sep 14, 2022
Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks Abstract Facial expression recognition in video

Bogireddy Sai Prasanna Teja Reddy 103 Dec 29, 2022
Neurolab is a simple and powerful Neural Network Library for Python

Neurolab Neurolab is a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework

152 Dec 06, 2022
MTCNN face detection implementation for TensorFlow, as a PIP package.

MTCNN Implementation of the MTCNN face detector for Keras in Python3.4+. It is written from scratch, using as a reference the implementation of MTCNN

Iván de Paz Centeno 1.9k Dec 30, 2022
⚓ Eurybia monitor model drift over time and securize model deployment with data validation

View Demo · Documentation · Medium article 🔍 Overview Eurybia is a Python library which aims to help in : Detecting data drift and model drift Valida

MAIF 172 Dec 27, 2022
CBREN: Convolutional Neural Networks for Constant Bit Rate Video Quality Enhancement

CBREN This is the Pytorch implementation for our IEEE TCSVT paper : CBREN: Convolutional Neural Networks for Constant Bit Rate Video Quality Enhanceme

Zhao Hengrun 3 Nov 04, 2022
Spatial Intention Maps for Multi-Agent Mobile Manipulation (ICRA 2021)

spatial-intention-maps This code release accompanies the following paper: Spatial Intention Maps for Multi-Agent Mobile Manipulation Jimmy Wu, Xingyua

Jimmy Wu 70 Jan 02, 2023
PyTorch Kafka Dataset: A definition of a dataset to get training data from Kafka.

PyTorch Kafka Dataset: A definition of a dataset to get training data from Kafka.

ERTIS Research Group 7 Aug 01, 2022
A system used to detect whether a person is wearing a medical mask or not.

Mask_Detection_System A system used to detect whether a person is wearing a medical mask or not. To open the program, please follow these steps: Make

Mohamed Emad 0 Nov 17, 2022
An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Decoupled-Contrastive-Learning This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper. Requirements P

Ramin Nakhli 71 Dec 04, 2022
Boosted neural network for tabular data

XBNet - Xtremely Boosted Network Boosted neural network for tabular data XBNet is an open source project which is built with PyTorch which tries to co

Tushar Sarkar 175 Jan 04, 2023
This project is for a Twitter bot that monitors a bird feeder in my backyard. Any detected birds are identified and posted to Twitter.

Backyard Birdbot Introduction This is a silly hobby project to use existing ML models to: Detect any birds sighted by a webcam Identify whic

Chi Young Moon 71 Dec 25, 2022
Learning Features with Parameter-Free Layers (ICLR 2022)

Learning Features with Parameter-Free Layers (ICLR 2022) Dongyoon Han, YoungJoon Yoo, Beomyoung Kim, Byeongho Heo | Paper NAVER AI Lab, NAVER CLOVA Up

NAVER AI 65 Dec 07, 2022
PyAF is an Open Source Python library for Automatic Time Series Forecasting built on top of popular pydata modules.

PyAF (Python Automatic Forecasting) PyAF is an Open Source Python library for Automatic Forecasting built on top of popular data science python module

CARME Antoine 405 Jan 02, 2023
Generating Digital Painting Lighting Effects via RGB-space Geometry (SIGGRAPH2020/TOG2020)

Project PaintingLight PaintingLight is a project conducted by the Style2Paints team, aimed at finding a method to manipulate the illumination in digit

651 Dec 29, 2022
Captcha-tensorflow - Image Captcha Solving Using TensorFlow and CNN Model. Accuracy 90%+

Captcha Solving Using TensorFlow Introduction Solve captcha using TensorFlow. Learn CNN and TensorFlow by a practical project. Follow the steps, run t

Jackon Yang 869 Jan 06, 2023
🕹️ Official Implementation of Conditional Motion In-betweening (CMIB) 🏃

Conditional Motion In-Betweening (CMIB) Official implementation of paper: Conditional Motion In-betweeening. Paper(arXiv) | Project Page | YouTube in-

Jihoon Kim 81 Dec 22, 2022