MLReef is an open source ML-Ops platform that helps you collaborate, reproduce and share your Machine Learning work with thousands of other users.

Overview

The collaboration platform for Machine Learning

MLReef is an open source ML-Ops platform that helps you collaborate, reproduce and share your Machine Learning work with thousands of other users.


MLReef

MLReef is a ML/DL development platform containing four main sections:

  • Data-Management - Fully versioned data hosting and processing infrastructure
  • Publishing code repositories - Containerized and versioned script repositories for immutable use in data pipelines
  • Experiment Manager - Experiment tracking, environments and results
  • ML-Ops - Pipelines & Orchestration solution for ML/DL jobs (K8s / Cloud / bare-metal)


To find out more about how MLReef can streamline your Machine Learning Development Lifecycle visit our homepage

Data Management

  • Host your data using git / git LFS repositories.
    • Work concurrently on data
    • Fully versioned or LFS version control
    • Full view on data processing and visualization history
  • Connect your external storage to MLReef and use your data directly in pipelines
  • Data set management (access, history, pipelines)

Publishing Code

Adding only parameter annotations to your code...

# example of parameter annotation for a image crop function
 @data_processor(
        name="Resnet50",
        author="MLReef",
        command="resnet50",
        type="ALGORITHM",
        description="CNN Model resnet50",
        visibility="PUBLIC",
        input_type="IMAGE",
        output_type="MODEL"
    )
    @parameter(name='input-path', type='str', required=True, defaultValue='train', description="input path")
    @parameter(name='output-path', type='str', required=True, defaultValue='output', description="output path")
    @parameter(name='height', type='int', required=True, defaultValue=224, description="height of cropped images in px")
    @parameter(name='width', type='int', required=True, defaultValue=224, description="width of cropped images in px")
    def init_params():
        pass

...and publishing your scripts gets you the following:

  • Containerization of your scripts
    • Always working scripts including easy hyperparameter access in pipelines
    • Execution environment (including specific packages & versions)
    • Hyper-parameters
      • ArgParser for command line parameters with currently used values
      • Explicit parameters dictionary
      • Input validation and guides
  • Multiple containers based on version and code branches

Experiment Manager

  • Complete experiment setup log
    • Full source control info including non-committed local changes
    • Execution environment (including specific packages & versions)
    • Hyper-parameters
  • Full experiment output automatic capture
    • Artifacts storage and standard-output logs
    • Performance metrics on individual experiments and comparative graphs for all experiments
    • Detailed view on logs and outputs generated
  • Extensive platform support and integrations

ML-Ops

  • Concurrent computing pipelining
  • Governance and control
    • Access and user management
    • Single permission management
    • Resource management
  • Model management

MLReef Architecture

The MLReef ML components within the ML life cycle:

  • Data Storage components based currently on Git and Git LFS.
  • Model development based on working modules (published by the community or your team), data management, data processing / data visualization / experiment pipeline on hosted or on-prem and model management.
  • ML-Ops orchestration, experiment and workflow reproducibility, and scalability.

Why MLReef?

MLReef is our solution to a problem we share with countless other researchers and developers in the machine learning/deep learning universe: Training production-grade deep learning models is a tangled process. MLReef tracks and controls the process by associating code version control, research projects, performance metrics, and model provenance.

We designed MLReef on best data science practices combined with the knowleged gained from DevOps and a deep focus on collaboration.

  • Use it on a daily basis to boost collaboration and visibility in your team
  • Create a job in the cloud from any code repository with a click of a button
  • Automate processes and create pipelines to collect your experimentation logs, outputs, and data
  • Make you ML life cycle transparent by cataloging it all on the MLReef platform

Getting Started as a Developer

To start developing, continue with the developer guide

Canonical source

The canonical source of MLReef where all development takes place is hosted on gitLab.com/mlreef/mlreef.

License

MIT License (see the License for more information)

Documentation, Community and Support

More information in the official documentation and on Youtube.

For examples and use cases, check these use cases or start the tutorial after registring:

If you have any questions: post on our Slack channel, or tag your questions on stackoverflow with 'mlreef' tag.

For feature requests or bug reports, please use GitLab issues.

Additionally, you can always reach out to us via [email protected]

Contributing

Merge Requests are always welcomed ❤️ See more details in the MLReef Contribution Guidelines.

Owner
MLReef
Your entire Machine Learning life cycle in one platform.
MLReef
Built various Machine Learning algorithms (Logistic Regression, Random Forest, KNN, Gradient Boosting and XGBoost. etc)

Built various Machine Learning algorithms (Logistic Regression, Random Forest, KNN, Gradient Boosting and XGBoost. etc). Structured a custom ensemble model and a neural network. Found a outperformed

Chris Yuan 1 Feb 06, 2022
2D fluid simulation implementation of Jos Stam paper on real-time fuild dynamics, including some suggested extensions.

Fluid Simulation Usage Download this repo and store it in your computer. Open a terminal and go to the root directory of this folder. Make sure you ha

Mariana Ávalos Arce 5 Dec 02, 2022
This project has Classification and Clustering done Via kNN and K-Means respectfully

This project has Classification and Clustering done Via kNN and K-Means respectfully. It later tests its efficiency via F1/accuracy/recall/precision for kNN and Davies-Bouldin Index for Clustering. T

Mohammad Ali Mustafa 0 Jan 20, 2022
A classification model capable of accurately predicting the price of secondhand cars

The purpose of this project is create a classification model capable of accurately predicting the price of secondhand cars. The data used for model building is open source and has been added to this

Akarsh Singh 2 Sep 13, 2022
Iterative stochastic gradient descent (SGD) linear regressor with regularization

SGD-Linear-Regressor Iterative stochastic gradient descent (SGD) linear regressor with regularization Dataset: Kaggle “Graduate Admission 2” https://w

Zechen Ma 1 Oct 29, 2021
MooGBT is a library for Multi-objective optimization in Gradient Boosted Trees.

MooGBT is a library for Multi-objective optimization in Gradient Boosted Trees. MooGBT optimizes for multiple objectives by defining constraints on sub-objective(s) along with a primary objective. Th

Swiggy 66 Dec 06, 2022
MLReef is an open source ML-Ops platform that helps you collaborate, reproduce and share your Machine Learning work with thousands of other users.

The collaboration platform for Machine Learning MLReef is an open source ML-Ops platform that helps you collaborate, reproduce and share your Machine

MLReef 1.4k Dec 27, 2022
Quantum Machine Learning

The Machine Learning package simply contains sample datasets at present. It has some classification algorithms such as QSVM and VQC (Variational Quantum Classifier), where this data can be used for e

Qiskit 364 Jan 08, 2023
The unified machine learning framework, enabling framework-agnostic functions, layers and libraries.

The unified machine learning framework, enabling framework-agnostic functions, layers and libraries. Contents Overview In a Nutshell Where Next? Overv

Ivy 8.2k Dec 31, 2022
A Python implementation of FastDTW

fastdtw Python implementation of FastDTW [1], which is an approximate Dynamic Time Warping (DTW) algorithm that provides optimal or near-optimal align

tanitter 651 Jan 04, 2023
Predico Disease Prediction system based on symptoms provided by patient- using Python-Django & Machine Learning

Predico Disease Prediction system based on symptoms provided by patient- using Python-Django & Machine Learning

Felix Daudi 1 Jan 06, 2022
Scikit-Garden or skgarden is a garden for Scikit-Learn compatible decision trees and forests.

Scikit-Garden or skgarden (pronounced as skarden) is a garden for Scikit-Learn compatible decision trees and forests.

260 Dec 21, 2022
Adaptive: parallel active learning of mathematical functions

adaptive Adaptive: parallel active learning of mathematical functions. adaptive is an open-source Python library designed to make adaptive parallel fu

741 Dec 27, 2022
Python package for machine learning for healthcare using a OMOP common data model

This library was developed in order to facilitate rapid prototyping in Python of predictive machine-learning models using longitudinal medical data from an OMOP CDM-standard database.

Sontag Lab 75 Jan 03, 2023
Decision tree is the most powerful and popular tool for classification and prediction

Diabetes Prediction Using Decision Tree Introduction Decision tree is the most powerful and popular tool for classification and prediction. A Decision

Arjun U 1 Jan 23, 2022
Regularization and Feature Selection in Least Squares Temporal Difference Learning

Regularization and Feature Selection in Least Squares Temporal Difference Learning Description This is Python implementations of Least Angle Regressio

Mina Parham 0 Jan 18, 2022
Turning images into '9-pan' palettes using KMeans clustering from sklearn.

img2palette Turning images into '9-pan' palettes using KMeans clustering from sklearn. Requirements We require: Pillow, for opening and processing ima

Samuel Vidovich 2 Jan 01, 2022
MLR - Machine Learning Research

Machine Learning Research 1. Project Topic 1.1. Exsiting research Benmark: https://paperswithcode.com/sota ACL anthology for NLP papers: http://www.ac

Charles 69 Oct 20, 2022
A benchmark of data-centric tasks from across the machine learning lifecycle.

A benchmark of data-centric tasks from across the machine learning lifecycle.

61 Dec 28, 2022
Open-Source CI/CD platform for ML teams. Deliver ML products, better & faster. ⚡️🧑‍🔧

Deliver ML products, better & faster Giskard is an Open-Source CI/CD platform for ML teams. Inspect ML models visually from your Python notebook 📗 Re

Giskard 335 Jan 04, 2023