A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.

Overview

Xcessiv

PyPI license PyPI Build Status

Xcessiv is a tool to help you create the biggest, craziest, and most excessive stacked ensembles you can think of.

Stacked ensembles are simple in theory. You combine the predictions of smaller models and feed those into another model. However, in practice, implementing them can be a major headache.

Xcessiv holds your hand through all the implementation details of creating and optimizing stacked ensembles so you're free to fully define only the things you care about.

The Xcessiv process

Define your base learners and performance metrics

define_base_learner

Keep track of hundreds of different model-hyperparameter combinations

list_base_learner

Effortlessly choose your base learners and create an ensemble with the click of a button

ensemble

Features

  • Fully define your data source, cross-validation process, relevant metrics, and base learners with Python code
  • Any model following the Scikit-learn API can be used as a base learner
  • Task queue based architecture lets you take full advantage of multiple cores and embarrassingly parallel hyperparameter searches
  • Direct integration with TPOT for automated pipeline construction
  • Automated hyperparameter search through Bayesian optimization
  • Easy management and comparison of hundreds of different model-hyperparameter combinations
  • Automatic saving of generated secondary meta-features
  • Stacked ensemble creation in a few clicks
  • Automated ensemble construction through greedy forward model selection
  • Export your stacked ensemble as a standalone Python file to support multiple levels of stacking

Installation and Documentation

You can find installation instructions and detailed documentation hosted here.

FAQ

Where does Xcessiv fit in the machine learning process?

Xcessiv fits in the model building part of the process after data preparation and feature engineering. At this point, there is no universally acknowledged way of determining which algorithm will work best for a particular dataset (see No Free Lunch Theorem), and while heuristic optimization methods do exist, things often break down into trial and error as you try to find the best model-hyperparameter combinations.

Stacking is an almost surefire method to improve performance beyond that of any single model, however, the complexity of proper implementation often makes it impractical to apply them in practice outside of Kaggle competitions. Xcessiv aims to make the construction of stacked ensembles as painless as possible and lower the barrier for entry.

I don't care about fancy stacked ensembles and what not, should I still use Xcessiv?

Absolutely! Even without the ensembling functionality, the sheer amount of utility provided by keeping track of the performance of hundreds, and even thousands of ML models and hyperparameter combinations is a huge boon.

How does Xcessiv generate meta-features for stacking?

You can choose whether to generate meta-features through cross-validation (stacked generalization) or with a holdout set (blending). You can read about these two methods and a lot more about stacked ensembles in the Kaggle Ensembling Guide. It's a great article and provides most of the inspiration for this project.

Contributing

Xcessiv is in its very early stages and needs the open-source community to guide it along.

There are many ways to contribute to Xcessiv. You could report a bug, suggest a feature, submit a pull request, improve documentation, and many more.

If you would like to contribute something, please visit our Contributor Guidelines.

Project Status

Xcessiv is currently in alpha and is unstable. Future versions are not guaranteed to be backwards-compatible with current project files.

Comments
  • Can't Use

    Can't Use

    Sorry for what is no doubt a stupid question:

    I've started Redis via redis-server. It says it's running on port 6379. Then I run xcessiv, but it takes me to a page that's not found. The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.. Any idea what I can do? I'm really eager to use Xcessiv.

    opened by xnmp 6
  • Automated ensembling techniques

    Automated ensembling techniques

    Working for a while with Xcessiv, I feel there's a need for some way to automate the selection of base learners in an ensemble. I'm unaware of existing techniques for this, so if anyone has any suggestions or could point me towards relevant literature, it would be greatly appreciated.

    enhancement 
    opened by reiinakano 5
  • Added more of the sklearn regressors to the presets

    Added more of the sklearn regressors to the presets

    Added the large majority of the more popular regressors of sklearn. I am aware that a few may be missing. Also, I tidied the code slightly and split the regressors and classifiers into two sections.

    opened by enisnazif 4
  • Memory management

    Memory management

    First of all, thanks! I find this project fascinating. My question/issue is about how do you handle the memory for multiple processes. By default Python will create a copy of the data per process. This is prohibitive for large datasets.

    How did you manage this problem?

    opened by alvarouc 3
  • Move .gitignore to project root and add Python ignores

    Move .gitignore to project root and add Python ignores

    I think the best practice for .gitignore is to have a single .gitignore file at the root of the project so I moved the .gitignore that was in xcessiv/ui (I think it was generated by create-react-scripts) to the project root and added some Python ignore lines.

    opened by menglewis 3
  • Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Crossvalidation to cvsetting.py

    Added Leave One Out Cross validation as part of #15

    I'm keen to finish implementing all of the cv / metrics within sklearn, just wanted to make sure I was doing it right since this is my first pull request!

    opened by enisnazif 2
  • XGBRegressor model stuck in queued status

    XGBRegressor model stuck in queued status

    I tried to make a regression model to run on zillow data from kaggle available here https://www.kaggle.com/c/zillow-prize-1/data Here is a gist of my dataset extractor as well as setting up the XGBRegressor and an exception that was the last thing left in the console https://gist.github.com/jef5ez/a9b0650293f343682a58b0f0500f3332 I selected the shuffle split for both cross validation settings and added MSE as the learner metric. The base learner seems to verify fine on the boston housing data. After hitting finalize and selecting a single base learner a row shows up below but is stuck in the Queued status.

    python 3.5.2 xcessiv (0.2.2) xgboost (0.6a2)

    opened by jef5ez 2
  • The _BasePipeline in exported Python script should be _BaseComposition

    The _BasePipeline in exported Python script should be _BaseComposition

    Since scitkit-learn 0.19.x, the base class for Pipeline has changed to _BaseComposition. https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py When using the generated code for training, it raises a name-not-found error on newer versions of sklearn. At the moment, an easy workaround is to change two instances of the word manually in the generated script.

    opened by Mithrillion 1
  • Issues with TfidnVectorizer

    Issues with TfidnVectorizer

    Hey, great tool.

    I have a problem though when I am trying to use a TfidfVectorizer for Text Classification. When I create a Single Base Learner I get the error:

    ValueError: all the input array dimensions except for the concatenation axis must match exactly .

    The type of the X variable is an numpy.ndarray, but if I don't convert the variable X to an array then I get the error message:

    TypeError: Singleton array array(<92820x194 sparse matrix of type '<class 'numpy.float64'>' with 92820 stored elements in Compressed Sparse Row format>, dtype=object) cannot be considered a valid collection.

    I choose the preset learner setting scikit-learn Random Forest as a Base Learner Type.

    import os
    import numpy as np
    import pandas as pd
    import pickle
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.ensemble import RandomForestClassifier
    
    def extract_main_dataset():
        # pandas data frame with the columns Classification, FeatureVector
        # ie:
        # 0, 'This is the feature vector'
        # 1, 'This is another feature vector' 
        # 2, 'This is yet another feature vector' 
        # 1, 'This is the last feature vector example' 
        with open('feature_vector.pik', 'rb') as rf:
            feature_vector = pickle.load(rf)
    
        y = np.array(feature_vector.Classification.values)
        title_rf_vectorizer = TfidfVectorizer(ngram_range=(2, 9),
                                              sublinear_tf=True,
                                              use_idf=True,
                                              strip_accents='ascii')
    
        title_rf_classifier = RandomForestClassifier(n_estimators=100, n_jobs=8)
        X = title_rf_vectorizer.fit_transform(feature_vector["Classification"]).toarray()
        return X, y
    
    opened by bbowler86 1
  • Valid values for metric to optimise in bayesian optimisation?

    Valid values for metric to optimise in bayesian optimisation?

    Is there a list of valid metric_to_optimise for Bayesian Optimisation?

    I am using sklearn mean_squared_regression for my base learning but when I enter that into the Bayesian Optimisation menu under metric_to_optimise I get:

    assert module.metric_to_optimize in automated_run.base_learner_origin.metric_generators
    AssertionError
    
    question 
    opened by Data-drone 1
  • 'dict_keys' object does not support indexing

    'dict_keys' object does not support indexing

    On lines 306 and 309 of views.py, trying to index a dictionary keys object will fail on Python 3 and result in a server error. The fix is simple: change all occurrences of

    base_learner_origin.validation_results.keys()[0]

    to

    list(base_learner_origin.validation_results.keys())[0]

    opened by KhaledSharif 1
  • redis.exceptions.DataError at xcessiv launch

    redis.exceptions.DataError at xcessiv launch

    Hello, When I try to launch xcessiv I get an error:

    Traceback (most recent call last): File "/PATH_TO/anaconda3/bin/xcessiv", line 10, in <module> sys.exit(main()) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/xcessiv/scripts/runapp.py", line 51, in main redis_conn.get(None) # will throw exception if Redis is unavailable File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 1264, in get return self.execute_command('GET', name) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/client.py", line 774, in execute_command connection.send_command(*args) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 620, in send_command self.send_packed_command(self.pack_command(*args)) File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 663, in pack_command for arg in imap(self.encoder.encode, args): File "/PATH_TO/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 125, in encode "byte, string or number first." % typename) redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a byte, string or number first.

    Previously I had to change from gevent.wsgi import WSGIServer to from gevent.pywsgi import WSGIServer as indicated in this issue

    My server is responding when I do redis-cli ping

    I am on Ubuntu 18.04, with python 3.7.3 and redis 5.0.5

    Do you have an idea to fix this? Thanks!

    opened by AlexCoul 0
  • How to import homemade modules in Xcessiv?

    How to import homemade modules in Xcessiv?

    I'm trying to import homemade module named preprocessing_115v (filename preprocessing_115v.py) into the main data extraction source code but I can't seem to find it :

    ############# import preprocessing_115v <-- where do I store the preprocessing_115v.py file for it to load here? def extract_main_dataset(): import pandas as pd df=pd.read_csv('./data.csv', sep=',',header=None) X=df.values labels=pd.read_csv('./labelsnum.csv', sep=',',header=None) y=labels.values y=y[:,0] return X, y ##############

    Amazing program by the way :-)

    opened by fcoppey 0
  • xcessiv server

    xcessiv server

    Hi This project looks very cool, but I am having some problems with the setup. I am running this in a container (my own), and I can't get the server to show up. From inside the container I can see the server running - ps shows xcessiv running and curl localhost:1994 gives me some HTML from xcessiv. From outside the container, however, there's nothing.

    I suppose that's down to the server.py file which I have now changed to this:

      1 from __future__ import absolute_import, print_function, division, unicode_literals¬                                                                              
      2 from gevent.wsgi import WSGIServer¬
      3 # import webbrowser¬
      4 ¬
      5 ¬
      6 def launch(app):¬
      7     http_server = WSGIServer(('0.0.0.0', app.config['XCESSIV_PORT']), app)¬
      8     # webbrowser.open_new('http://localhost:' + str(app.config['XCESSIV_PORT']))¬
      9     http_server.serve_forever()¬
    

    I have changed the WSGIServer setup to be open to outside connection (I suppose that's what I changed), but it's still not showing up.

    Feedback appreciated. I'd like to try this out. Thanks!

    opened by benman1 0
  • Feature Request - Backup .db file

    Feature Request - Backup .db file

    I got an error something to the effect of "Error with JSON "N" at position 8345", presumably caused by my manually editing the code for one of the base learners. Once I got this error however, none of the base learners in my project would load. I resolved it by manually deleting the base learner I had been editing from the .db file. I'll post the specifics if I can recreate it, but I'm wondering if it might be prudent to have some kind of db backup/"Last Known Good Configuration"?

    opened by Tahlor 0
  • Fix issue #63 no module named wsgi

    Fix issue #63 no module named wsgi

    In file server.py

    from gevent.wsgi import WSGIServer
    

    Has to be changed to:

    from gevent.pywsgi import WSGIServer
    

    http://www.gevent.org/api/gevent.pywsgi.html

    opened by KhaledTo 0
  • ImportError: No module named wsgi

    ImportError: No module named wsgi

    File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xcessiv/server.py", line 2, in from gevent.wsgi import WSGIServer ImportError: No module named wsgi

    opened by xialeizhou 2
Releases(v0.5.1)
Owner
Reiichiro Nakano
I like working on awesome things with awesome people!
Reiichiro Nakano
3D-Reconstruction 基于深度学习方法的单目多视图三维重建

基于深度学习方法的单目多视图三维重建 Part I 三维重建 代码:Part1 技术文档:[Markdown] [PDF] 原始图像:Original Images 点云结果:Point Cloud Results-1

HMT_Curo 19 Dec 26, 2022
BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work

BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work. For this project, I used the sigmoid function as an activation

Manas Bommakanti 1 Jan 22, 2022
RDA: Robust Domain Adaptation via Fourier Adversarial Attacking

RDA: Robust Domain Adaptation via Fourier Adversarial Attacking Updates 08/2021: check out our domain adaptation for video segmentation paper Domain A

17 Nov 30, 2022
Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory

Approximate Outer Product Gradient Descent with Memory Code for the numerical experiment of the paper Speeding-Up Back-Propagation in DNN: Approximate

2 Mar 02, 2022
PyTorch code for Composing Partial Differential Equations with Physics-Aware Neural Networks

FInite volume Neural Network (FINN) This repository contains the PyTorch code for models, training, and testing, and Python code for data generation t

Cognitive Modeling 20 Dec 18, 2022
A scanpy extension to analyse single-cell TCR and BCR data.

Scirpy: A Scanpy extension for analyzing single-cell immune-cell receptor sequencing data Scirpy is a scalable python-toolkit to analyse T cell recept

ICBI 145 Jan 03, 2023
Train an imgs.ai model on your own dataset

imgs.ai is a fast, dataset-agnostic, deep visual search engine for digital art history based on neural network embeddings.

Fabian Offert 5 Dec 21, 2021
Code for CVPR2021 paper "Learning Salient Boundary Feature for Anchor-free Temporal Action Localization"

AFSD: Learning Salient Boundary Feature for Anchor-free Temporal Action Localization This is an official implementation in PyTorch of AFSD. Our paper

Tencent YouTu Research 146 Dec 24, 2022
Minimisation of a negative log likelihood fit to extract the lifetime of the D^0 meson (MNLL2ELDM)

Minimisation of a negative log likelihood fit to extract the lifetime of the D^0 meson (MNLL2ELDM) Introduction The average lifetime of the $D^{0}$ me

Son Gyo Jung 1 Dec 17, 2021
Source code for "Roto-translated Local Coordinate Framesfor Interacting Dynamical Systems"

Roto-translated Local Coordinate Frames for Interacting Dynamical Systems Source code for Roto-translated Local Coordinate Frames for Interacting Dyna

Miltiadis Kofinas 19 Nov 27, 2022
The implementation of the CVPR2021 paper "Structure-Aware Face Clustering on a Large-Scale Graph with 10^7 Nodes"

STAR-FC This code is the implementation for the CVPR 2021 paper "Structure-Aware Face Clustering on a Large-Scale Graph with 10^7 Nodes" 🌟 🌟 . 🎓 Re

Shuai Shen 87 Dec 28, 2022
Optimizers-visualized - Visualization of different optimizers on local minimas and saddle points.

Optimizers Visualized Visualization of how different optimizers handle mathematical functions for optimization. Contents Installation Usage Functions

Gautam J 1 Jan 01, 2022
Morphable Detector for Object Detection on Demand

Morphable Detector for Object Detection on Demand (ICCV 2021) PyTorch implementation of the paper Morphable Detector for Object Detection on Demand. I

9 Feb 23, 2022
This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021.

PyTorch implementation of DAQ This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021. For more informatio

CV Lab @ Yonsei University 36 Nov 04, 2022
A library of scripts that interact with the PythonTurtle module to create games, drawings, and more

TurtleLib TurtleLib is a library of scripts that interact with the PythonTurtle module to create games, drawings, and more! Using the Scripts Copy or

1 Jan 15, 2022
Chainer implementation of recent GAN variants

Chainer-GAN-lib This repository collects chainer implementation of state-of-the-art GAN algorithms. These codes are evaluated with the inception score

399 Oct 23, 2022
Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORAL)

Scribble-Supervised LiDAR Semantic Segmentation Dataset and code release for the paper Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORA

102 Dec 25, 2022
Unofficial PyTorch implementation of "RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving" (ECCV 2020)

RTM3D-PyTorch The PyTorch Implementation of the paper: RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (ECCV 2020

Nguyen Mau Dzung 271 Nov 29, 2022
An easy-to-use app to visualise attentions of various VQA models.

Ask Me Anything: A tool for visualising Visual Question Answering (AMA) An easy-to-use app to visualise attentions of various VQA models. Please click

Apoorve 37 Nov 13, 2022
In this project, we'll be making our own screen recorder in Python using some libraries.

Screen Recorder in Python Project Description: In this project, we'll be making our own screen recorder in Python using some libraries. Requirements:

Hassan Shahzad 4 Jan 24, 2022