The command line interface for Gradient - Gradient is an an end-to-end MLOps platform

Overview

GitHubSplash

Gradient CLI

PyPI Downloads


Get started: Create AccountInstall CLITutorialsDocs

Resources: WebsiteBlogSupportContact Sales


Gradient is an an end-to-end MLOps platform that enables individuals and organizations to quickly develop, train, and deploy Deep Learning models. The Gradient software stack runs on any infrastructure e.g. AWS, GCP, on-premise and low-cost Paperspace GPUs. Leverage automatic versioning, distributed training, built-in graphs & metrics, hyperparameter search, GradientCI, 1-click Jupyter Notebooks, our Python SDK, and more.

Key components:

  • Notebooks: 1-click Jupyter Notebooks.
  • Workflows: Train models at scale with composable actions.
  • Inference: Deploy models as API endpoints.

Gradient supports any ML/DL framework (TensorFlow, PyTorch, XGBoost, etc).


See releasenotes.md for details on the current release, as well as release history.


Getting Started

  1. Make sure you have a Paperspace account set up. Go to http://paperspace.com to register and generate an API key.

  2. Use pip, pipenv, or conda to install the gradient package, e.g.:

    pip install -U gradient

    To install/update prerelease (Alpha/Beta) version version of gradient, use:

    pip install -U --pre gradient

  3. Set your api key by executing the following:

    gradient apiKey

    Note: your api key is cached in ~/.paperspace/config.json

    You can remove your cached api key by executing:

    gradient logout

Executing tasks on Gradient

The Gradient CLI follows a standard [command] [--options] syntax

For example, to create a new Deployment use:

gradient workflows create [type] [--options]

For a full list of available commands run gradient workflows --help. You can also view more info about Workflows in the docs.

Contributing

Want to contribute? Contact us at [email protected]

Pre-Release Testing

Have a Paperspace QA tester install your change directly from the branch to test it. They can do it with pip install git+https://github.com/Paperspace/[email protected].

Comments
  • Some parameters in job_client.create() don't work

    Some parameters in job_client.create() don't work

    I'm following the sample docs at gradient.api_sdk.clients.job_client. I can't get working_directory or job_env to work.

    working_directory: auto set to /paperspace. When I pass in /app, it's ignored & /paperspace is still used (listed Console > Job > Environment; and on the Job model fetch from job_client.list()). I just cashed in and set all my Docker stuff from /app to /paperspace and it worked. Ideally I wouldn't even want to pass /app, but it'd just be picked up from WORKDIR in the Dockerfile (which is /app)

    job_env: I can't find a workaround (save putting a config.json in /storage). Parameter gets gets ignored; of note, the docs has a typo:

    job = job_client.create(
        ...
        job_env={
            'CUSTOM_ENV'='Some value that will be set as system environment',
        }
    )
    

    'CUSTOM_ENV'='..' should be 'CUSTOM_ENV': '..'. Not nit-picking; just that it had me second guessing my approach, since using a dict gets ignored ([j.job_env for j in job_client.list()] => [None, None, ..]) so I tried json.dumps(my_dict), still ignored. Any suggestions on getting the job_env passed in?

    Overall, it seems like some params are respected, and others not; and it's hard to know which is which. Maybe a mismatch on the docs vs Github vs PyPi?

    opened by lefnire 8
  • Unable to install due to numpy==1.19.4

    Unable to install due to numpy==1.19.4

    Hi I am unable to use gradient on python 3.7.9 due to RuntimeError caused by numpy==1.19.4

    RuntimeError: The current Numpy installation ('c:\users\XXXX\envs\gradient\lib\site-packages\numpy\init.py') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86

    I tried downgrading the numpy, but it appears to be incompatible and i got the ff ModuleNotFoundError: ModuleNotFoundError: No module named '_curses'

    opened by thompsonalecbgo 7
  • Update Readme.md

    Update Readme.md

    I found two outdated or currently not working commands while trying to execute my container:

    1. The --project option was ignored but providing --projectId worked.
    2. The example for the --command option had as error message: Error: Missing argument "SCRIPT...". Upon providing a script the originally provided command was ignored. Instead I had to use the --shell option to execute the correct call.
    opened by nziermann 6
  • Keep artifacts for canceled jobs

    Keep artifacts for canceled jobs

    I am aware this is probably not the place to open this issue, but it seems like artifacts are not kept when a job is canceled. But why? I just finished a 2 days running job and wanted to have access to my best model, but the artifacts are not showing up potentially because I canceled the job. If this is the case, this is very frustrating.

    opened by MichelML 5
  • feat(notebooks): enable basic notebook lifecycle commands PS-13680

    feat(notebooks): enable basic notebook lifecycle commands PS-13680

    Newly Supported Commands

    • notebooks stop
    • notebook start
    • notebook create
    • notebook fork
    • notebook artifacts list

    Related tickets

    https://paperspace.atlassian.net/browse/PS-12219 https://paperspace.atlassian.net/browse/PS-11559 https://paperspace.atlassian.net/browse/PS-13681 https://paperspace.atlassian.net/browse/PS-13682 https://paperspace.atlassian.net/browse/PS-13683 https://paperspace.atlassian.net/browse/PS-13684

    This PR follows up on the PR for PS-12219 (https://github.com/Paperspace/PS_API/pull/1526). It enables createNotebook, startNotebook, artifactsList and forkNotebook commands with the v2 endpoint. It adds in the ability for creating a notebook with vm_type_id or vm_type_label

    opened by kevin-kabore 4
  • Failing to start notebook

    Failing to start notebook

    I'm trying to start an instance of an existing notebook: gradient notebooks start --id [id] --machineType Free-P5000

    and get in return: Failed to create resource: Cluster null not found

    specifying any cluster ID doesn't change anything and the output remains the same.

    A bug ?

    bug 
    opened by macsunmood 3
  • CR2-22 CR2-49 CR2-48 add metrics list to sdk

    CR2-22 CR2-49 CR2-48 add metrics list to sdk

    Adding list custom metrics functionality to sdk and cli for jobs, experiments, deployments, and notebooks

    QA Test Plan:

    1. in cli, check that this new list command shows up in the help text: gradient deployments metrics --help
    2. create a deployment (gui or cli doesn't matter) and wait for it to finish provisioning
    3. try the new list command: gradient deployments metrics list --id XYZ, should return custom metrics (a list of words),
    4. repeat 1 and 2 for jobs, experiments, and notebooks. these might return null but that's okay for now, just checking that this new command is available. note the stuff that returns null below if any and I'll solve that in a separate ticket.
    released 
    opened by robghchen 3
  • Fix: Serialize jobenv to envVars PS-15020

    Fix: Serialize jobenv to envVars PS-15020

    Fixes serialization of job environment

    Test Plan: Create a job and specify jobEnv parameter. Use command "env" to see if job environment is affected

    released 
    opened by paperspace-philip 3
  • When providing --ignoreFiles comma separated the code returns Attribute error.

    When providing --ignoreFiles comma separated the code returns Attribute error.

    The command below with the --ignoreFiles field set

    gradient experiments run single node \
    --name test \
    --projectid 12345 \
    --container paperspace/tensorflow-python \
    --machineType V100 \
    --command 'python main.py' \
    --ignoreFiles "file1,file2,file3,folder1,folder2,folder3"
    

    Returns the following error

    file_paths = self._retrieve_file_paths(workspace_path, ignore_files)
    File "/lib/python3.7/site-packages/gradient/workspace.py", line 59, in _retrieve_file_paths
        exclude += ignored_files.split(',')
    AttributeError: 'list' object has no attribute 'split'
    
    opened by EvDuijnhoven 3
  • Notebook example with SDK concepts

    Notebook example with SDK concepts

    Example of using new SDK functionality to create projects, experiments, analyze model, and create an inferrence deployment.

    Tested on TensorFlow 2.0 & Python3 container image.

    opened by dte 3
  • Replace faulty chunk counting logic

    Replace faulty chunk counting logic

    PR #384 for allowing multipart uploads introduced a bug that prevents datasets from being uploaded that contain files that are multiples of 500Mb. This became more of an issue with #389 which changes the chunk sizes to 15Mb, meaning that any file that is a multiple of 15Mb (75Mb in my case) will cause the dataset to fail to upload.

    What I believe is happening is that because we faulty logic instructs us to read an additional block of data from the filesystem that doesn't exist. How this appears in my experience is that the CLI hangs indefinitely, or in a Workflow it crashes.

    released 
    opened by fmorlock-tt 2
  • Failed to execute request against storage provider when uploading a dataset

    Failed to execute request against storage provider when uploading a dataset

    Dear everyone,

    I am trying to upload a new version of my dataset, but I keep getting Brokenpipe errors. Here is what I am doing :

    gradient datasets versions create --id ...
    gradient datasets files put --id ...:... --source-path "."
    
    Failed to execute request against storage provider: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
    

    Uploading to other cloud providers is working, but not here.

    By the way, is gradient-cli uploading everything again at each new version? Or is it comparing hashes to avoid unnecessary file transfers?

    Thanks, Clément

    opened by clementpoiret 0
Releases(v2.0.6)
Owner
Paperspace
Paperspace
Helicopter animation in terminal

helicopter-helicopter Helicopter animation in terminal (scroll down for instructions) Why does this exist? It's because of a meme Click for details Se

Wasi Master 7 Mar 14, 2022
A dec-bin converter uses 2's complement.

2's Complement Dec-Bin Converter A dec-bin converter uses 2's complement. Visit my Medium Post. What is 2's complement? Two's complement is the most c

C.H Jacky 9 Mar 01, 2022
Python commandline tool for remembering linux/terminal commands

ehh Remember linux commands Commandline tool for remembering linux/terminal commands. It stores your favorite commands in ~/ehh.json in your homedir a

56 Nov 10, 2022
Interact with Replit remotely with the Replit CLI

Replit CLI pip install repl-cli Welcome to Replit CLI! With the Replit CLI Application, you can work with your repls locally, including clone, pull,

Shuchir Jain 4 Aug 18, 2022
Tarstats - A simple Python commandline application that collects statistics about tarfiles

A simple Python commandline application that collects statistics about tarfiles.

Kristian Koehntopp 13 Feb 20, 2022
A VIM-inspired filemanager for the console

ranger 1.9.3 ranger is a console file manager with VI key bindings. It provides a minimalistic and nice curses interface with a view on the directory

12.6k Dec 30, 2022
Declarative CLIs with argparse and dataclasses

argparse_dataclass Declarative CLIs with argparse and dataclasses. Features Features marked with a ✓ are currently implemented; features marked with a

Mike DePalatis 29 Dec 06, 2022
Hurry is a CLI tool to speed setting up MoniGoMani HyperStrategy & co. #freqtrade #hyperopting #trading #strategy

Hurry is a CLI tool to speed setting up MoniGoMani HyperStrategy & co. #freqtrade #hyperopting #trading #strategy

10 Dec 29, 2022
A simple CLI application helps you to find giant files that are eating up your system storage

Large file finder Sometimes it's very hard to find if some giant files are eating up your system storage. We might need to hunt those down. This simpl

Rahul Baruri 5 Nov 18, 2022
The WalletsNet CLI helps you connect to WalletsNet

WalletsNet CLI The WalletsNet CLI helps you connect to WalletsNet. With the CLI, you can: Trigger webhook events or resend events for easy testing Tai

WalletsClub 8 Dec 22, 2021
PyDropper - pick colors everywhere

PyDropper - pick colors everywhere Downloads Settings PyDropper is an eyedropper

Herman Brunberg 2 Jan 04, 2022
Cli tool to browse and play anime

browse and watch anime (scrape from gogoanime) (wip) basically ani-cli but in python cuz python good demo dependencies mpv installation from pypi pip

sheep padowo 2 Apr 20, 2022
CLI tool to develop StarkNet projects written in Cairo

OpenZeppelin Nile ⛵ Navigate your StarkNet projects written in Cairo. Getting started Create a folder for your project and cd into it: mkdir myproject

OpenZeppelin 305 Dec 30, 2022
Automaton - python script to execute bash command based on changes in size of a file.

automaton python script to execute given command = everytime size of a given file changes,hence everytime a file is modified.(almost) download automa

asrar bhat 1 Jan 03, 2022
The Prisma Cloud CLI is a command line interface for Prisma Cloud by Palo Alto Networks.

Prisma Cloud CLI The Prisma Cloud CLI is a command line interface for Prisma Cloud by Palo Alto Networks. Support This project has been developed by P

Palo Alto Networks 13 Oct 14, 2022
A simple command line virtual operating system, written in python

Virtual operating system A simple virtual operating system written in python. (Under development). Currently, the following commands are supported: Co

B.Jothin kumar 7 Nov 15, 2022
Simple CLI prompt for easy I/O with OpenAI's API

openai-cli-prompt Simple CLI prompt for easy I/O with OpenAI's API Quickstart Create a .env file with: OPENAI_API_KEY=Your OpenAI API Key Configure

Erik Nomitch 1 Oct 12, 2021
A command line tool to publish ads on ebay-kleinanzeigen.de

kleinanzeigen-bot Feedback and high-quality pull requests are highly welcome! About Installation Usage Development Notes License About kleinanzeigen-b

83 Dec 26, 2022
Python codecs extension featuring CLI tools for encoding/decoding anything

CodExt Encode/decode anything. This library extends the native codecs library (namely for adding new custom encodings and character mappings) and prov

Alex 210 Dec 30, 2022
Display Images in your terminal with python

Term-Img Display Images in your terminal with python NOTE: This project is a work in progress and not everything on here has actually been implemented

My avatar ;D 118 Jan 05, 2023