Describe the bug I'm trying to run the example notebooks (in codeflare/notebooks), and came across this error. The error persisted thru attempts to restart my kernel, entire machine, and re-cloning the repo. Any help, or an explanation of the root cause, is much appreciated!

To Reproduce Steps to reproduce the behavior:

1. Go to notebooks/plot_nca_classification.ipynb
2. Run 2nd code block. It uses Ray and Codeflare.
3. This line produces the error knn_pipeline = rt.select_pipeline(pipeline_fitted, pipeline_fitted.get_xyrefs(node_knn)[0])
4. See error: RaySystemError: System error: buffer source array is read-only

Full stack trace:

RaySystemError: System error: buffer source array is read-only
traceback: Traceback (most recent call last):
  File "/home/kastan/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/serialization.py", line 268, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/home/kastan/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/serialization.py", line 191, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/home/kastan/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/serialization.py", line 169, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/home/kastan/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/serialization.py", line 157, in _deserialize_pickle5_data
    obj = pickle.loads(in_band, buffers=buffers)
  File "sklearn/neighbors/_dist_metrics.pyx", line 223, in sklearn.neighbors._dist_metrics.DistanceMetric.__setstate__
  File "stringsource", line 658, in View.MemoryView.memoryview_cwrapper
  File "stringsource", line 349, in View.MemoryView.memoryview.__cinit__
ValueError: buffer source array is read-only


---------------------------------------------------------------------------
RaySystemError                            Traceback (most recent call last)
/tmp/ipykernel_1251/3313313255.py in <module>
      9 test_input.add_xy_arg(node_scalar, dm.Xy(X_test, y_test))
     10 
---> 11 knn_pipeline = rt.select_pipeline(pipeline_fitted, pipeline_fitted.get_xyrefs(node_knn)[0])
     12 knn_score = ray.get(rt.execute_pipeline(knn_pipeline, ExecutionType.SCORE, test_input)
     13                     .get_xyrefs(node_knn)[0].get_yref())

~/.pyenv/versions/3.8.6/lib/python3.8/site-packages/codeflare/pipelines/Runtime.py in select_pipeline(pipeline_output, chosen_xyref)
    381         curr_xyref = xyref_queue.get()
    382         curr_node_state_ptr = curr_xyref.get_curr_node_state_ref()
--> 383         curr_node = ray.get(curr_node_state_ptr)
    384         prev_xyrefs = curr_xyref.get_prev_xyrefs()
    385 

~/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/_private/client_mode_hook.py in wrapper(*args, **kwargs)
     87             if func.__name__ != "init" or is_client_mode_enabled_by_default:
     88                 return getattr(ray, func.__name__)(*args, **kwargs)
---> 89         return func(*args, **kwargs)
     90 
     91     return wrapper

~/.pyenv/versions/3.8.6/lib/python3.8/site-packages/ray/worker.py in get(object_refs, timeout)
   1621                     raise value.as_instanceof_cause()
   1622                 else:
-> 1623                     raise value
   1624 
   1625         if is_individual_id:

Expected behavior Expected is selecting the pipeline and evaluating its score via a 'SCORE' pipeline.

Desktop

• OS: Ubuntu 20.04 via WSL2 on Windows.
• Python 3.8.6

Thank you for any help! I am a University of Illinois at Urbana-Champaign grad student trying to make the most of your work!

Overview

Currently, lineage uses SimpleQueue to realize pipelines. But this is available only in Python >=3.8. This reduces adoption, moving to Queue will give us broader Python version coverage.

Acceptance Criteria

• [x] Replace SimpleQueue with Queue
• [x] Ensure tests pass

Questions

• What are the drawbacks of using Queue vs SimpleQueue?

Assumptions

Reference

• https://towardsdatascience.com/dive-into-queue-module-in-python-its-more-than-fifo-ce86c40944ef

What does this PR do? This is first step towards improving resiliency and performance in Ray without modifying the source code. This PR includes a new tool that helps configure Ray cluster conveniently. The tool helps in fetching and parsing ray configurations, and generating resiliency profiles (e.g., strict, relaxed, recommended). Currently, we are working on deciding configuration options for each resiliency profile manually by evaluating them on various ray workloads. We'll update this PR accordingly.

Description of Changes The changes in this PR is currently independent of the main codeFlare code. We intend to put this tool in a new folder called utils in the codeFlare root directory.

Describe the bug After running a PREDICT, the y_predcannot be obtained via get_yref(), instead can be obtained via the get_Xref(). Semantically, this seems weird.

To Reproduce Steps to reproduce the behavior:

1. Go to https://github.ibm.com/codeflare/ray-pipeline/blob/complex-example-1/notebooks/plot_feature_selection_pipeline.ipynb
2. Scroll down to `y_pred = ray.get(predict_clf_output[0].get_yref())
3. If you change that statement to y_pred = ray.get(predict_clf_output[0].get_Xref()) the output would match the original sklearn pipeline on the top.

Expected behavior The predicted output should be obtained from calling get_yref().

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

Describe the bug sample pipeline jupyter notebook errors out due to undefined variable

To Reproduce Steps to reproduce the behavior:

1. Go to binder
2. Click on sample pipeline jupyter notebook
3. Run

Expected behavior Jupyter notebook on binder should run without exception

Additional context error while executing cell:

pipeline_output = rt.execute_pipeline(pipeline, ExecutionType.FIT, pipeline_input)
node_0_output = pipeline_output.get_xyrefs(node_0)


In [74]:

outputs[0]



---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-74-a45df8d4a457> in <module>
----> 1 outputs[0]

NameError: name 'outputs' is not defined

Describe the bug Jupyter notebook kernel dies

To Reproduce Steps to reproduce the behavior:

1. Go to Binder
2. Click on plot_scalable_poly_kernels
3. Run the notebook

Expected behavior The jupyter notebook should run without error

Graph viz missing from binder service

To Reproduce Steps to reproduce the behavior:

1. Go to binder service 2.Run Grid search notebook

Additional context

Below error caused by execution of cell:

non_param_graph = cf_utils.pipeline_to_graph(pipeline)
non_param_graph

ExecutableNotFound: failed to execute ['dot', '-Kdot', '-Tsvg'], make sure the Graphviz executables are on your systems' PATH

Related PRs Extending #37

What does this PR do?

This PR extends the Ray resiliency config tool. The PR does the following:

1. The Ray config utility script now supports configurations from Ray v1.6, 1.7, and 1.8.

2. The tool now saves config. files into their respective version directory. This is more organized as compared to saving files from all Ray versions into a single folder. For example, now the tools save output config. files in the following manner by default. ├── configs │ ├── 1.0.0 │ │ ├── Ray 1.0.0 related config files │ ├── 1.1.0 │ │ └── Ray 1.1.0 related config files

3. The configuration parsing code is more generalized than before. Removed some hard-coded conditions and added functions to make the code less cluttered.

4. Added a new field called config_string in the output config file. This field stores the original string from which we parsed the default value of the configuration. The config_string stores string whenever the default value is not a simple value but a conditional statement. This field will help in explaining how the associated environment variable's value will determine the default value. For example: For raylet_start_wait_time_s configuration, the signature/input is following:

RAY_CONFIG(uint32_t, raylet_start_wait_time_s,
           std::getenv("RAY_preallocate_plasma_memory") != nullptr &&
                   std::getenv("RAY_preallocate_plasma_memory") == std::string("1")
               ? 120
               : 10)

And, the script dumps following Yaml entry in the .conf file:

raylet_start_wait_time_s:
  config_string: 'std::getenv("RAY_preallocate_plasma_memory") != nullptr && std::getenv("RAY_preallocate_plasma_memory") == std::string("1") ? 120 : 10'
  default: '10'
  env: RAY_preallocate_plasma_memory
  type: uint32_t
  value_for_this_mode: '10'

The new field i.e. config_string is informatory and gives an idea about how the associated environment variable will be processed.

5. The config tool now uses YAML format variable instead of a hardcoded string for system-config map YAML (system_cm.yaml)

Related Issue

Supports #27

Related PRs

Reopen PR 31 after PR 27 merged with develop.

What does this PR do?

Description of Changes Checked node exists in pipeline post_graph Added ExecutionType.TRANSFORM Added a unit test

Related Issue

Supports #22

Related PRs

This PR is not dependent on any other PR

What does this PR do?

Description of Changes

Assign PREDICT and SCORE results to yref as appropriate in Runtime.py. Updated unit tests and notebook examples.

What gif most accurately describes how I feel towards this PR?

Describe the bug Possibly a corner case? ray-pipeline/codeflare/pipelines/Datamodel.py in get_pre_edges(self, node) 640 """ 641 pre_edges = [] --> 642 pre_nodes = self.pre_graph[node] 643 # Empty pre 644 if not pre_nodes:

KeyError: <codeflare.pipelines.Datamodel.EstimatorNode object at 0x7fa2d8920f10>

To Reproduce

## initialize codeflare pipeline by first creating the nodes
pipeline = dm.Pipeline()
node_a = dm.EstimatorNode('a', MinMaxScaler())
node_b = dm.EstimatorNode('b', StandardScaler())
node_c = dm.EstimatorNode('c', MaxAbsScaler())
node_d = dm.EstimatorNode('d', RobustScaler())

node_e = dm.AndNode('e', FeatureUnion())
node_f = dm.AndNode('f', FeatureUnion())
node_g = dm.AndNode('g', FeatureUnion())

## codeflare nodes are then connected by edges
pipeline.add_edge(node_a, node_e)
pipeline.add_edge(node_b, node_e)
pipeline.add_edge(node_c, node_f)
## node_d does not have a downstream node
# pipeline.add_edge(node_d, node_f)
pipeline.add_edge(node_e, node_g)
pipeline.add_edge(node_f, node_g)

pipeline_input = dm.PipelineInput()
xy = dm.Xy(X,y)
pipeline_input.add_xy_arg(node_a, xy)
pipeline_input.add_xy_arg(node_b, xy)
pipeline_input.add_xy_arg(node_c, xy)
pipeline_input.add_xy_arg(node_d, xy)

## execute the codeflare pipeline
pipeline_output = rt.execute_pipeline(pipeline, ExecutionType.FIT, pipeline_input)

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

Describe the bug Cannot bring up Ray cluster as defined in the OCP tutorial

To Reproduce Steps to reproduce the behavior:

1. Go to https://codeflare.readthedocs.io/en/latest/getting_started/starting.html#Openshift-Ray-Cluster-Operator
2. Run pip3 install --upgrade codeflare
3. Create namespace oc create namespace codeflare
4. Run ray up ray/python/ray/autoscaler/kubernetes/example-full.yaml fails:

$ ray up ray/python/ray/autoscaler/kubernetes/example-full.yaml
Provided cluster configuration file (ray/python/ray/autoscaler/kubernetes/example-full.yaml) does not exist

Expected behavior Bring up Ray cluster on OCP

Desktop (please complete the following information):

• OS: MacOS

Additional context OCP Cluster running on IBM Cloud.

$ oc cluster-info
Kubernetes master is running at https://c100-e.jp-tok.containers.cloud.ibm.com:31129

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

CodeFlare commit hash commit a2b290a115b0cc1317270cef6059d5281215842e

Overview

As a CFP user, I would like to split a dataset (e.g., np array, pandas dataframe) into smaller objects that can then be fed into other nodes/pipeline. This is especially useful when we have compute intensive tasks and would like to parallelize it easily.

Acceptance Criteria

• [x] Design for splitter, should be simple and intuitive
• [ ] Implementation as an extension to the Node construct
• [x] Tests

Questions

• What type of semantics does the splitter node define?

Assumptions

Reference

Overview

As a Codeflare user, I want to use Ray and Spark alternately to execute my end-to-end ML jobs. Some steps might be executed more efficiently using Ray, while others using Spark. The plasma store in Ray seems to provide an efficient way to share ObjectRef between Ray and Spark. Currently, RayDP project supports from Spark to Ray in some limited way, by running Spark as a Ray actor. However, ObjectRef cannot be shared easily in both directions, Spark-2-Ray and Ray-2-Spark.

Acceptance Criteria

• Pandas dataframe created by remote tasks in local Ray plasma stores can be passed with ObjectRefto the Spark driver to create a Spark dataframecontaining list of ObjectRef.
• Once that is done, on the Spark side, the executors of Spark can then access to the original Pandas dataframe locally.
• From Spark to Ray: Spark preserves groupby() partition semantics and writes these partitions to plasma store, instead of using hashPartition().

Questions

• In RayDP, only the driver node knows about and can access Ray. The executors of PySpark doesn't have access to Ray. This will prevent the PySpark executors from accessing the Ray plasma store. As a result, it is not possible to seamlessly pass ObjectRefbetween Ray workers and Spark executors.

Assumptions

• Ray and Spark can share data seamlessly by exchanging ObjectRef among Ray workers and Spark executors.

Reference

[Reference] I have opened an issue on the RayDP repo: https://github.com/oap-project/raydp/issues/164

Overview

As a CF pipelines user, support for nested pipelines, where the node of a pipeline can be a pipeline itself.

Acceptance Criteria

• [ ] Nested pipeline API
• [ ] Nested pipeline implementation
• [ ] ADR for supporting nested pipelines
• [ ] Tests

Questions

• Given that pipelines are not estimators by themselves, how can we support nesting easily?

Assumptions

Reference

Overview

As a CF pipelines user, I would like to understand the memory consumption when pipelines are executed. Given pipelines accept nparrays, will zero copy sharing of Ray help?

Acceptance Criteria

• [ ] Memory growth as pipelines are executed
• [ ] Clear documentation on this
• [ ] A potential story explaining this in more detail

Questions

Assumptions

Reference

Overview

As a CF pipelines user, I would like the ability to select the best or k-best pipelines from a parameter grid search output.

Acceptance Criteria

• [ ] Best pipeline selection
• [ ] K-best pipeline selection
• [ ] Tests and compatibility with sklearn outputs

Questions

Assumptions

Reference