Replacing BaselineClassifier with DummyClassifier from Sklearn. Relacing BaselineRegressor with DummyRegressor from Sklearn. Created new operator with make_operator with schemas defined in BaselineClassifier and BaselineRegressor

Contributed by students of SRH University, Heidelberg. https://github.com/tauseefhashmi https://github.com/frankcode101 https://github.com/tanmaygaikwad https://github.com/RajathReddy9 https://github.com/vickyvishal/

• BaggingClassifier - Add 2 constraints.
• BaggingRegressor - Add 2 constraints.
• ColumnTransformer - Add a sparse constraint.
• ExtraTreesClassifier - Add a constraint.
• ExtraTreesRegressor - Add 2 constraints.
• FeatureAgglomeration - Add a sparse constraint.
• FunctionTransformer - Fix a constraint.
• GaussianNB - Add a sparse constraint.
• KNeighborsClassifier - Remove 2 constraints (false negatives).
• KNeighborsRegressor - Remove 2 constraints (false negatives).
  • For KNN-C and KNN-R, this constraint might be useful but the schema is long and contains many TODOs.

Metric 'minkowski' not valid for sparse input. Use sorted(sklearn.neighbors.VALID_METRICS_SPARSE['brute']) to get valid options. Metric can also be a callable function.

• LinearSVC - Improve constaints.
• LinearSVR - Modify "loss" schema. Add a constraint.
• LogisticRegression - Add 4 constaints.
• MinMaxScaler - Add a sparse constraint.
• MissingIndicator - Add a constraint.
• OneHotEncoder - Add "drop" schema, new in 0.21. Add a constraint.
• OrdinalEncoder - Remove "ignore" from "handle_unknown" schema. Add 2 constraints.
• RandomForestClassifier - Add a constraint.
• RandomForestRegressor - Add 2 constraints.
• RidgeClassifier - Add 2 constraints.
• Ridge - Add 2 constraints.
• RobustScaler - Add a constraint.
• SimpleImputer - Add a constraint.
• SVC - Remove a constraint (false negative). Add a constraint.
• SVR - Remove a constraint (false negative). Add a constraint.

isort module observed on the outer scripts. So that neither search/*.py, lib/*.py, datasets/*.py, and utils/*.py have been changed which means that the schema2search_space.py file is still like the usual.

Created additional Logisticaix360 , added notebook comparing the Prejudice remover|Logistic regression| Logisticaix360 notebook and added testcases .

IBM Watson Studio:Version 1.1.0-151 (1.1.0-151) on macOS Catalina 10.15.4

from sklearn.preprocessing import Normalizer
from sklearn.tree import DecitionTreeRegressor as Tree
from lale.lib.lale import Hyperopt


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-28-2eee442a0b4d> in <module>
----> 1 from lale.lib.lale import Hyperopt

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/lale/lib/lale/__init__.py in <module>
     61 from .baseline_classifier import BaselineClassifier
     62 from .baseline_regressor import BaselineRegressor
---> 63 from .grid_search_cv import GridSearchCV
     64 from .hyperopt import Hyperopt
     65 from .topk_voting_classifier import TopKVotingClassifier

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/lale/lib/lale/grid_search_cv.py in <module>
     15 from typing import Any, Dict
     16 
---> 17 import lale.lib.sklearn
     18 import lale.search.lale_grid_search_cv
     19 import lale.operators

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/lale/lib/sklearn/__init__.py in <module>
    130 from .extra_trees_classifier import ExtraTreesClassifier
    131 from .extra_trees_regressor import ExtraTreesRegressor
--> 132 from .feature_agglomeration import FeatureAgglomeration
    133 from .function_transformer import FunctionTransformer
    134 from .gaussian_nb import GaussianNB

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/lale/lib/sklearn/feature_agglomeration.py in <module>
     13 # limitations under the License.
     14 
---> 15 import sklearn.cluster.hierarchical
     16 import lale.docstrings
     17 import lale.operators

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/sklearn/cluster/__init__.py in <module>
      4 """
      5 
----> 6 from .spectral import spectral_clustering, SpectralClustering
      7 from .mean_shift_ import (mean_shift, MeanShift,
      8                           estimate_bandwidth, get_bin_seeds)

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/sklearn/cluster/spectral.py in <module>
     15 from ..metrics.pairwise import pairwise_kernels
     16 from ..neighbors import kneighbors_graph
---> 17 from ..manifold import spectral_embedding
     18 from .k_means_ import k_means
     19 

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/sklearn/manifold/__init__.py in <module>
      3 """
      4 
----> 5 from .locally_linear import locally_linear_embedding, LocallyLinearEmbedding
      6 from .isomap import Isomap
      7 from .mds import MDS, smacof

~/WatsonStudioDesktop/miniconda3/envs/desktop/lib/python3.6/site-packages/sklearn/manifold/locally_linear.py in <module>
     10 from scipy.sparse.linalg import eigsh
     11 
---> 12 from ..base import BaseEstimator, TransformerMixin, _UnstableArchMixin
     13 from ..utils import check_random_state, check_array
     14 from ..utils.extmath import stable_cumsum

ImportError: cannot import name '_UnstableArchMixin'

Replacing BaselineClassifier with DummyClassifier from sklearn. Relacing BaselineRegressor with DummyRegressor from sklearn.

Contributed by students of SRH University, Heidelberg. https://github.com/tauseefhashmi https://github.com/frankcode101 https://github.com/tanmaygaikwad https://github.com/RajathReddy9 https://github.com/vickyvishal/

Our lale.lib.lale package has a BaselineClassifier that simply predicts the majority class. Scikit-learn has a DummyClassifier that does the same, with a few additional useful configuration options. We should add the DummyClassifier to our lale.lib.sklearn package, and eliminate the BaselineClassifier, since it is redundant. Similarly, we should also replace lale.lib.lale.BaselineRegressor by scikit-learn's DummyRegressor.

Hi All,

I have implemented a custom imputer based on Scikit-learn SimpleImputer as an example. My code lives in albert_imputer.py. Everything in fine until the final result is being printed. This is what I see in debugger:

> /Users/albert/miniconda3/envs/lale/lib/python3.7/site-packages/lale/pretty_print.py(160)_get_module_name()
-> op = find_op(mod_name_short, op_name)
(Pdb) l
155  	    mod_name_long = class_name[: class_name.rfind(".")]
156  	    mod_name_short = mod_name_long[: mod_name_long.rfind(".")]
157  	    unqualified = class_name[class_name.rfind(".") + 1 :]
158  	    if class_name.startswith("lale.") and unqualified.endswith("Impl"):
159  	        unqualified = unqualified[: -len("Impl")]
160  ->	    op = find_op(mod_name_short, op_name)
161  	    if op is not None:
162  	        mod = mod_name_short
163  	    else:
164  	        op = find_op(mod_name_long, op_name)
165  	        if op is not None:
(Pdb) p mod_name_long, mod_name_short, unqualified,
('albert_imputer', 'albert_impute', 'AlbertImputerImpl')

In "mod_name_short" the last "r" is missed. For this reason, importlib cannot load the module in find_op(). As a temporary workaround, I created a symbolic link "albert_impute.py" to "albert_imputer.py" file and it works.

Inside autoai_core, estimators are identified by their class name (rather than a full path include the module etc). While there does not seem to be any limitation on the Lale side related to having multiple estimators with the same class name, it looks like significant changes would be required in AutoAI core in order to handle this, and these changes would propagate all the way up to the top-level interface.

In order to have smooth integration without changing the KaggleBot interface, I propose to simply prefix the snapml esimator class names with Snap. I've tested these changes with autoai_core development branch and everything seems to work nicely.

Fixes #596

In the existing implementation of fair_stratified_train_test_split, there is no argument for setting random_state and we internally just set its value to 42 when calling scikit learn's train_test_split. It will be good idea to add this argument to the fair_stratified_train_test_split method similar to corresponding scikit routine.

Not actually an issue, just a question.

Hyperopt supports Random Search, Tree of Parzen Estimators (TPE), and Adaptive TPE.

When using optimizer Hyperopt in lale, which is the search algorithm behind?

Thank you in advance for your time and contribution.

There are 43 operators that exist in both lale.lib.autogen and lale.lib.sklearn. This overlap is problematic, because it can lead to unexpected behavior depending on the order of imports, and users might end up with a lower-quality version of an operator for which we also have a higher-quality version. We should simply remove lale.lib.autogen operators from the repository for which there is also a lale.lib.sklearn operator. To avoid breaking code that uses them, we can change the __init__.py file of lale.lib.autogen to forward to the relevant replacements.

List of duplicate operators: ada_boost_classifier, ada_boost_regressor, decision_tree_classifier, decision_tree_regressor, extra_trees_classifier, extra_trees_regressor, function_transformer, gaussian_nb, gradient_boosting_classifier, gradient_boosting_regressor, isomap, k_means, k_neighbors_classifier, k_neighbors_regressor, linear_regression, linear_svc, linear_svr, logistic_regression, min_max_scaler, missing_indicator, mlp_classifier, multinomial_nb, nmf, normalizer, nystroem, one_hot_encoder, ordinal_encoder, passive_aggressive_classifier, pca, polynomial_features, quadratic_discriminant_analysis, quantile_transformer, random_forest_classifier, random_forest_regressor, ridge, ridge_classifier, robust_scaler, sgd_classifier, sgd_regressor, simple_imputer, standard_scaler, svc, svr

There is an existing ad-hoc set of tests for our our optimizers.
We should factor out a standard set of tests that are generic over the choice of optimizer, that can be re-used/run against each optimizer. This can then be used as a test suite for new optimizers, including ones that live in other repositories (such as NSGA-II based optimizer recently added to lale-gpl)

It would be nice if the user could provide a pipeline with more preprocessing subpipelines than necessary. For example, if a pipeline contains a branch with one-hot encoding for string columns, but the data only has numeric columns, it would be convenient if it worked anyway. Unfortunately, some sklearn operators raise an exception when their input data has zero columns. This issue proposes preventing that exception during fit, and possibly even pruning them from the pipeline returned by fit.

Example:

import sklearn.datasets
X, y = sklearn.datasets.load_digits(return_X_y=True)

from lale.lib.lale import Project, ConcatFeatures
from lale.lib.sklearn import LogisticRegression, OneHotEncoder

proj_nums = Project(columns={"type": "number"})
proj_cats = Project(columns={"type": "string"})
one_hot = OneHotEncoder(handle_unknown="ignore")
prep = (proj_nums & (proj_cats >> one_hot)) >> ConcatFeatures
trainable = prep >> LogisticRegression()

print(f"shapes: X {X.shape}, y {y.shape}, "
      f"nums {proj_nums.fit(X).transform(X).shape}, "
      f"cats {proj_cats.fit(X).transform(X).shape}")

trained = trainable.fit(X, y)

This prints:

shapes: X (1797, 64), y (1797,), nums (1797, 64), cats (1797, 0)
Traceback (most recent call last):
  File "~/tmp.py", line 17, in <module>
    trained = trainable.fit(X, y)
  File "~/git/user/lale/lale/operators.py", line 3981, in fit
    trained = trainable.fit(X=inputs)
  File "~/git/user/lale/lale/operators.py", line 2526, in fit
    trained_impl = trainable_impl.fit(X, y, **filtered_fit_params)
  File "~/git/user/lale/lale/lib/sklearn/one_hot_encoder.py", line 145, in fit
    self._wrapped_model.fit(X, y)
  File "~/python3.7venv/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 385, in fit
    self._fit(X, handle_unknown=self.handle_unknown)
  File "~/python3.7venv/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 74, in _fit
    X_list, n_samples, n_features = self._check_X(X)
  File "~/python3.7venv/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 43, in _check_X
    X_temp = check_array(X, dtype=None)
  File "~/python3.7venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 72, in inner_f
    return f(**kwargs)
  File "~/python3.7venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 661, in check_array
    context))
ValueError: Found array with 0 feature(s) (shape=(1797, 0)) while a minimum of 1 is required.

Hyperopt 0.2.6 was released on November 15: https://pypi.org/project/hyperopt/0.2.6/

Unfortunately, it breaks many Lale tests: https://github.com/IBM/lale/actions/runs/1467468837

For example, the failures include some very basic tests such as:

• test.test_core_transformers.TestFeaturePreprocessing.test_MinMaxScaler
• test.test_core_transformers.TestFeaturePreprocessing.test_PCA
• test.test_core_transformers.TestConcatFeatures.test_concat_with_hyperopt

So for now, we limit the Hyperopt version to <=0.2.5: https://github.com/IBM/lale/pull/875/commits/24db05830ff79d0d1b474b5595a612bad9e62959

We should try to update to the latest (in fact, if we are lucky, Hyperopt 0.2.7 fixes the problem).

Add a test case to test_autoai_output_consumption.py covering the following scenario:

1. Read an output AutoAI pipeline.
2. Use DisparateImpactRemover on the preprocessing prefix and perform refinement with a choice of classifiers.
3. Use Hyperopt to choose the best model with the pre-estimator mitigation of step 2.

Here is some code for using the pipeline generated for the German credit dataset:

fairness_info = {
            "protected_attributes": [
                {"feature": "Sex", "reference_group": ['male'], "monitored_group": ['female']},
                {"feature": "Age", "reference_group": [[20,40], [60,90]], "monitored_group": [[41, 59]]}
            ],
            "favorable_labels": ["No Risk"],
            "unfavorable_labels": ["Risk"],
}

prefix = best_pipeline.remove_last().freeze_trainable()

from sklearn.linear_model import LogisticRegression as LR
from sklearn.ensemble import RandomForestClassifier as RF
from lale.operator_wrapper import wrap_imported_operators
from lale.lib.aif360 import DisparateImpactRemover
wrap_imported_operators()

di_remover = DisparateImpactRemover(**fairness_info, preparation=prefix, redact=True)
planned_fairer = di_remover >> (LR | RF)

from lale.lib.aif360 import accuracy_and_disparate_impact
from lale.lib.aif360 import FairStratifiedKFold

combined_scorer = accuracy_and_disparate_impact(**fairness_info)
fair_cv = FairStratifiedKFold(**fairness_info, n_splits=3)

from lale.lib.lale import Hyperopt

import pandas as pd
df = pd.read_csv("german_credit_data_biased_training.csv")
y = df.iloc[:, -1]
X = df.drop(columns=['Risk'])

trained_fairer = planned_fairer.auto_configure(
    X, y, optimizer=Hyperopt, cv=fair_cv, verbose=True,
    max_evals=1, scoring=combined_scorer, best_score=1.0)