The framework supports multiple execution backends: Ray, Dask, MPI and MultiProcessing.
To run tests you need to install pytest package:
pip install pytest
Run tests using command:
pytest unidist/test/
For unidist MPI backend follow additional instructions.
Signed-off-by: Igoshev, Yaroslav [email protected]
flake8 unidist/black unidist/git commit -s docs/developer/architecture.rst is up-to-date Signed-off-by: Igoshev, Yaroslav [email protected]
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date PR adds CLI interface for unidist. The examples of usage can be received:
user$: python unidist/cli -h
or if package is installed
user$: unidist --help
Signed-off-by: Alexey Prutskov [email protected]
Signed-off-by: Alexey Prutskov [email protected]
PR adds .readthedocs.yaml configuration file. Only one difference from the main doc building flow is using python3.8 instead of using 3.7 by default. In case of using python3.7 we have documentation built failed because MPI backend in this case requires optional pickle5 package. Updating python version to 3.8 is a way to fix this.
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Signed-off-by: Alexey Prutskov [email protected]
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Add logo to the README
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Signed-off-by: Alexey Prutskov [email protected]
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Unidist can't start mpi backend on Windows with the next exception:
C:\Users\aprutsko\repositories\unidist>python unidist\cli -m pytest unidist\test\test_general.py -b mpi
Windows fatal exception: access violation
Current thread 0x00005c44 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1174 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\communication.py", line 18 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\executor.py", line 13 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\__init__.py", line 5 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\backend.py", line 10 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\core\base\utils.py", line 88 in get_backend_proxy
File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 34 in make_decorator
File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 118 in remote
File "C:\Users\aprutsko\repositories\unidist\unidist\test\utils.py", line 40 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 843 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "C:\Users\aprutsko\repositories\unidist\unidist\test\test_general.py", line 10 in <module>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\assertion\rewrite.py", line 170 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
File "C:\Users\aprutsko\.conda\envs\unidist\lib\importlib\__init__.py", line 127 in import_module
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\pathlib.py", line 524 in import_path
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 578 in _importtestmodule
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 500 in _getobj
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 291 in obj
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 516 in _inject_setup_module_fixture
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 503 in collect
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in <lambda>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 311 in from_call
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in pytest_make_collect_report
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 458 in collect_one_node
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 808 in genitems
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 634 in perform_collect
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 333 in pytest_collection
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 322 in _main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 269 in wrap_session
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 162 in main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main
File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pytest\__main__.py", line 5 in <module>
File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 87 in _run_code
File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 194 in _run_module_as_main
===================================================================================
plasticc: | | I_MPI 112 previous | I_MPI 112 | MPICH 112 | |----------|--------------------|-----------|-----------| | read_csv | 187,14 | 185.85539 | | | t_etl | 735,44 | 380.46509 | |
ny_taxi: | | I_MPI 112 previous | I_MPI 112 | MPICH 112 | |----------|--------------------|-----------|-----------| | read_csv | 289,68 | 238.94134 | | | Q1 | 1,84 | 1.09677 | | | Q2 | 52,93 | 50.96434 | | | Q3 | 12,20 | 4.78096 | | | Q4 | 7,33 | 5.09975 | |
census: | | I_MPI 112 previous | I_MPI 112 | MPICH 112 | |----------|--------------------|-----------|-----------| | read_csv | 7,73 | 6.54384 | | | t_etl | 103,75 | 81.01938 | |
Modin plasticc benchmark reveals heavy contention problem when using all CPU cores and MPI backend. One heavy worker task contents with unidist.get receive operation, which previously never went to sleep.
Signed-off-by: Igoshev, Yaroslav [email protected]
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Signed-off-by: Igoshev, Iaroslav [email protected]
flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date flake8 .black .git commit -s docs/developer/architecture.rst is up-to-date Look at benefits and possible data transfer protocol problems for MPI backend if we use shared object storage for each node.
We should add sections/pages on execution flows of unidist with different backends:
This release contains multiple important bugfixes for MPI backend, which makes it be more functional and stable.
@YarShev @Retribution98
Source code(tar.gz)This release adds support for the dynamic spawning of MPI worker processes, contains significant refactoring of MPI backend, cut off unidist CLI, includes important updates for documentation and a bunch of bugfixes.
@YarShev, @prutskov, @no-ponomarev
Source code(tar.gz)This is the first release of undist, which includes support for different execution backends such as Ray, MPI, Dask, Python Multiprocessing, as well as a sequential Python backend for debugging. This release contains significant changes and upgrades to codebase, unidist's documentation, and some bugfixes.
Parameter (#63)@YarShev, @prutskov, @no-ponomarev
Source code(tar.gz)Euler_021_Py Euler Problem 021 solved in Python Let d(n) be defined as the sum o
Calc Interpreter An interpreter modeled after a calculator implemented in Python 3. The program currently only supports basic mathematical expressions
100 days of Python Following the initiative of my friend Helber Belmiro, who is almost done with his 100 days of Java, I have decided to start my 100
PyDrop An Airdrop alternative for cross-platform users only for desktop with Python, -version 1.0 with less effort, just as a practice. ##############
Author Tyler Windemuth Collatz_Conjecture A simple program to run through inputs for a 3n+1 problem Purpose: doesn't really have a purpose, did this t
NFT Generator This script generates NFT art based on its desired traits with their specific rarities. It has been used to generate the full collection
fbcek FBChecker Account using python , package requests and web old facebook using python 3.x apt upgrade -y apt update -y pkg install bash -y pkg ins
Subcert Subcert is a subdomain enumeration tool, that finds all the valid subdomains from certificate transparency logs. Table of contents Setup Demo
Datamol is a python library to work with molecules. It's a layer built on top of RDKit and aims to be as light as possible.
opevcvdl-hw2 This project uses openCV and Qt to achieve the requirements. Version Python 3.7 opencv-contrib-python 3.4.2.17 Matplotlib 3.1.1 pyqt5 5.1
By: Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans | sq Shqiptare Albania
JOTS Journal of the Session A webapp for taking fast notes, designed for business, school, and collaboration with groups.
What is this? Script used to perform automatic certificate unpinning of an APK by adding a custom network security configuration that permits user-add
Pytheas Main repository for the Pytheas submersible probe system. List of Acronyms/Terms USP - Underwater Sensor Platform - The primary platform in th
executing This mini-package lets you get information about what a frame is currently doing, particularly the AST node being executed. Usage Getting th
AlexaUsingPython - Alexa will pay attention to your order, as: Hello Alexa, play music, Hello Alexa, what's the time? Alexa will pay attention to your order, get it, and afterward do some activity as
Clean Commit From fixed RAnDom CRashes to [FIX] Fixed random crashes. Clean commit helps you by auto-formating your commits to make your repos better
Fisherman is a free open source fishing bot written in python.
Advent-of-code-2021 My attempt at this years Advent of Code! day 1: ** day 2: ** day 3: ** day 4: ** day 5: ** day 6: ** day 7: ** day 8: * day 9: day
Software for working Banking service 😄 This project was developed for Banking service. mysql server is required To have mysql server on your system u
1 Jan 24, 2022
1 Oct 17, 2021
19 Nov 08, 2021
6 Mar 25, 2022
0 Apr 22, 2022
24 Oct 08, 2022
5 Dec 24, 2022
59 Dec 16, 2022
276 Dec 19, 2022
3 Aug 17, 2022
1 Aug 26, 2022
2 Jun 10, 2022
5 Jul 28, 2021
2 Nov 19, 2022
211 Jan 01, 2023
10 Aug 18, 2022
3 Dec 26, 2021
33 Jan 29, 2022
1 Jul 06, 2022
1 Oct 21, 2021