The sarge package provides a wrapper for subprocess which provides command pipeline functionality.

Related tags

Documentationsarge
Overview

Overview

The sarge package provides a wrapper for subprocess which provides command pipeline functionality.

This package leverages subprocess to provide easy-to-use cross-platform command pipelines with a Posix flavour: you can have chains of commands using ;, &, pipes using | and |&, and redirection.

Requirements & Installation

The sarge package requires Python 2.6 or greater, and can be installed with the standard Python installation procedure:

python setup.py install

There is a set of unit tests which you can invoke with:

python setup.py test

before running the installation.

Availability & Documentation

The latest version of sarge can be found on GitHub.

The latest documentation (kept updated between releases) is on Read The Docs:

Please report any problems or suggestions for improvement either via the mailing list or the issue tracker.

Comments
  • problem with run async=true & returncodes  .....

    problem with run async=true & returncodes .....

    Hi

    below code always return [0, 0, None] and command never finish : seems have problem with sleep ...

    cmd = 'echo "Hello " && sleep 2  && echo "Finish " '
    p = run(cmd, async_=True)
    
    opened by Maziar123 19
  • hang with redirection

    hang with redirection

    Original report by Anonymous.


    When using redirection that leads to an ioerror, sarge can hang waiting for spawned processes that will never complete due to waiting on a full pipe that no one is reading.

    A simple test case:

    #!python
    
    import sarge
    pipeline = sarge.run('head -c 65K /dev/zero | cat > /does/not/exist')
    
    

    The first command hangs on a write to the pipe, which is never closed and sarge hangs on a wait() for this process that will never finish.

    bug major 
    opened by vsajip 13
  • Retcode is 0 using shell=True

    Retcode is 0 using shell=True

    Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).


    In [1]: from sarge import run
    
    In [2]: p = run('exit 1', shell=True)
    
    In [3]: p.returncode
    Out[3]: 0
    
    bug major 
    opened by vsajip 11
  • ValueError: need more than 2 values to unpack

    ValueError: need more than 2 values to unpack

    Original report by Allan Lei (Bitbucket: allanlei, GitHub: allanlei).


    I get this error for any commands I run.

    • Windows 7 Pro 64
    • Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
    • pip install sarge==0.1.2

    Simple run

    #!python
    
    sarge.run('echo abc')
    

    Trying the test_sarge.py script from repo

    #!bash
    
    PS C:\Users\allan.lei\Desktop> python test_sarge.py
    E.EEEEE.E...EEE..EEEEEEEE....EException in thread Thread-3:
    Traceback (most recent call last):
      File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
        self.run()
      File "C:\Python27\lib\threading.py", line 763, in run
        self.__target(*self.__args, **self.__kwargs)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
        result = getattr(self, method)(node, input, async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1201, in run_command_node
        node.cmd.run(input=input, async=async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 593, in run
        self.process = p = Popen(self.args, **self.kwargs)
      File "C:\Python27\lib\subprocess.py", line 701, in __init__
        errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 481, in _get_handles
        stderr)
    ValueError: need more than 2 values to unpack
    
    Exception in thread Thread-4:
    Traceback (most recent call last):
      File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
        self.run()
      File "C:\Python27\lib\threading.py", line 763, in run
        self.__target(*self.__args, **self.__kwargs)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
        result = getattr(self, method)(node, input, async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1289, in run_list_node
        self.run_node(curr, input, async=use_async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
        result = getattr(self, method)(node, input, async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1201, in run_command_node
        node.cmd.run(input=input, async=async)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 593, in run
        self.process = p = Popen(self.args, **self.kwargs)
      File "C:\Python27\lib\subprocess.py", line 701, in __init__
        errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
      File "C:\Python27\lib\site-packages\sarge\__init__.py", line 481, in _get_handles
        stderr)
    ValueError: need more than 2 values to unpack
    
    bug major 
    opened by vsajip 11
  • shell_quote(

    shell_quote("'ab?'") should return '"\'ab?\'"', not "'ab?'"

    Original report by David Lowe (Bitbucket: [David D Lowe](https://bitbucket.org/David D Lowe), ).


    Current behaviour:

    >>> shell_quote("'ab?'")
     "'ab?'"
    >>> print(shell_quote("'ab?'"))
    'ab?'
    

    Expected behaviour:

     >>> shell_quote("'ab?'")
     '"\'ab?\'"'
     >>> print(shell_quote("'ab?'"))
     "'ab?'"
    

    For example, to create a file named 'ab?', that is, a filename that consists of a single quote character, the a character, the b character, the ? character and the single quote character, in bash you would type the following:

     $ touch "'ab?'"
    

    And not the following:

     $ touch 'ab?'
    
    bug major 
    opened by vsajip 11
  • Use capture_both with Feeder

    Use capture_both with Feeder

    Original report by Jeet Ray (Bitbucket: shadowrylander, GitHub: shadowrylander).


    Hello! Would it be possible to use the feeder with capture_both? I would like to store the result in a variable until I can display or return it manually! My current code is something like this:

    from sarge import Feeder, get_stdout, capture_both
    
    feeder = Feeder()
    for line in capture_both("cowsay", input=feeder, async_=True).stdout:
        print(line.decode("utf-8").rstrip())
    feeder.feed(get_stdout("fortune"))
    feeder.close()
    

    Thank you kindly for the help!

    enhancement minor 
    opened by vsajip 9
  • Unstable/inconsistent behaviour of progress.py example code

    Unstable/inconsistent behaviour of progress.py example code

    Original report by CDuv (Bitbucket: CDuv, GitHub: CDuv).


    In want to use the sarge library to execute a simple cmd1 | cmd2 shell command from my Python script an grab it's output (both stdout and stderr) live while the shell command executes. I was previously using subprocess.Popen() + subprocess.PIPE + communicate() but the output was buffered.

    As a working base I tested the progress.py code detailed in the tutorial.

    File "test_sarge_live.py":

    import optparse # because of 2.6 support
    import sys
    import threading
    import time
    import logging
    
    from sarge import capture_stdout
    
    logging.basicConfig(level=logging.DEBUG, filename='/tmp/test_sarge_live.log',
                        filemode='w', format='%(asctime)s %(threadName)-10s %(name)-15s %(lineno)4d %(message)s')
    
    def progress(capture, options):
        lines_seen = 0
        messages = {
            'line 25\n': 'Getting going ...\n',
            'line 50\n': 'Well on the way ...\n',
            'line 75\n': 'Almost there ...\n',
        }
        while True:
            s = capture.readline()
            if not s and lines_seen:
                break
            if options.dots:
                sys.stderr.write('.')
            else:
                msg = messages.get(s)
                if msg:
                    sys.stderr.write(msg)
            lines_seen += 1
        if options.dots:
            sys.stderr.write('\n')
        sys.stderr.write('Done - %d lines seen.\n' % lines_seen)
    
    def main():
        parser = optparse.OptionParser()
        parser.add_option('-n', '--no-dots', dest='dots', default=True,
                          action='store_false', help='Show dots for progress')
        options, args = parser.parse_args()
    
        #~ p = capture_stdout('ncat -k -l -p 42421', async=True)
        p = capture_stdout('python lister.py -d 0.1 -c 100', async=True)
    
        t = threading.Thread(target=progress, args=(p.stdout, options))
        t.start()
    
        while(p.returncodes[0] is None):
            # We could do other useful work here. If we have no useful
            # work to do here, we can call readline() and process it
            # directly in this loop, instead of creating a thread to do it in.
            p.commands[0].poll()
            time.sleep(0.05)
        t.join()
    
    if __name__ == '__main__':
        sys.exit(main())
    

    But running it gives very inconsistent output:

    On Ubuntu v16.04 using Python v2.7.12 and sarge v0.1.4 I get theses (at random):

    • This ('NoneType' object has no attribute 'returncode' + sys.excepthook is missing)
    Traceback (most recent call last):
    File "test_sarge_live.py", line 55, in <module>
      sys.exit(main())
    File "test_sarge_live.py", line 46, in main
      while(p.returncodes[0] is None):
    File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1072, in returncodes
      result = [c.process.returncode for c in self.commands]
    AttributeError: 'NoneType' object has no attribute 'returncode'
    ..
    Done - 2 lines seen.
    close failed in file object destructor:                                                                                                                                                                                                                         
    sys.excepthook is missing
    lost sys.stderr
    
    • Or also this (no 'NoneType' object has no attribute 'returncode')
    close failed in file object destructor:
    sys.excepthook is missing
    lost sys.stderr
    
    • Or even this:
    Traceback (most recent call last):
    .  File "test_sarge_live.py", line 55, in <module>
      
    sys.exit(main())Done - 1 lines seen.
    
    File "test_sarge_live.py", line 46, in main
      while(p.returncodes[0] is None):
    IndexError: list index out of range
    Exception in thread Thread-1 (most likely raised during interpreter shutdown):
    Traceback (most recent call last):
    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    File "/usr/lib/python2.7/threading.py", line 754, in run
    File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
    File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
    File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
    File "/usr/lib/python2.7/threading.py", line 585, in set
    File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
    <type 'exceptions.TypeError'>: 'NoneType' object is not callable
    
    • And sometimes this no failure output:
    .
    Done - 1 lines seen.
    

    I also tested on Docker container Debian v9.4 using Python v2.7.13 and sarge v0.1.4: I get the same outputs.

    When I kill (via [Ctrl]+[C]) one instance and immediately run the script again I get:

    .Traceback (most recent call last):
      File "test_sarge_live.py", line 55, in <module>
        sys.exit(main())
      File "test_sarge_live.py", line 46, in main
        while(p.returncodes[0] is None):
    IndexError: list index out of range
    
    Done - 1 lines seen.
    Exception in thread Thread-1 (most likely raised during interpreter shutdown):
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
      File "/usr/lib/python2.7/threading.py", line 754, in run
      File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
      File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
      File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
      File "/usr/lib/python2.7/threading.py", line 585, in set
      File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
    <type 'exceptions.TypeError'>: 'NoneType' object is not callable
    [email protected]:/app# close failed in file object destructor:
    sys.excepthook is missing
    lost sys.stderr
    

    but I think this is normal.

    Is the tutorial code still accurate?

    bug minor 
    opened by vsajip 9
  • Inconsistent behavior with async=True

    Inconsistent behavior with async=True

    Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).


    Hi,

    I found that I can't specify sleep n call before any other command (including another sleep n call) without waiting for it using async=True. Is it expected behavior?

    This works as expected:

    In [1]: from sarge import run
    
    In [2]: p = run('echo abc | cat && sleep 5', async=True)
    # Returns immediately
    abc
    
    In [3]: p.commands[-1].poll()
    
    In [4]: p.commands[-1].poll()
    Out[4]: 0
    
    

    This is not:

    In [5]: p = run('sleep 5 && echo abc | cat', async=True)
    # Hangs for 5 seconds
    abc
    
    
    bug major 
    opened by vsajip 9
  • Capturing output from an infinite cycle

    Capturing output from an infinite cycle

    Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).


    Hi,

    I tried to run a simple script: run_forever.py

    import time
    
    i = 0
    while True:
        time.sleep(1)
        print(i)
        i += 1
    

    with sarge as described in the docs:

    In [1]: import sarge
    
    In [2]: p = sarge.run('python run_forever.py', async=True, stdout=sarge.Capture())
    

    However, I can't capture the output:

    In [3]: p.stdout.readline()
    Out[3]: b''
    
    In [4]: p.commands[0].stdout.readline()
    Out[4]: b''
    

    Am I doing it wrong? How can I achieve that?

    bug major 
    opened by vsajip 8
  • `shell_format` work incorrect on windows.

    `shell_format` work incorrect on windows.

    Original report by pahaz NA (Bitbucket: pahaz, GitHub: pahaz).


    from sarge import shell_shlex, shell_format
    import shlex
    import subprocess
    
    cmd = shell_format("python -c {0}", "print('aw')")
    print(cmd)
    

    This code produse:

    C:\Users\pahaz_000>C:\Python27\python.exe z.py
    python -c 'print('\''aw'\'')'
    

    And on windows whis in incorrect:

    C:\Users\pahaz_000\PycharmProjects\dokku>python -c 'print('\''aw'\'')'
      File "<string>", line 1
        'print('\''aw'\'')'
                          ^
    SyntaxError: unexpected character after line continuation character
    

    On linux correct:

    (venv)[email protected]:/vagrant# python -c 'print('\''aw'\'')'
    aw
    
    bug major 
    opened by vsajip 7
  • Child process stdout seems to be getting closed unexpectedly

    Child process stdout seems to be getting closed unexpectedly

    Original report by Paul Moore (Bitbucket: pmoore, GitHub: pmoore).

    The original report had attachments: yada.py, yadatest.py


    I wanted to use sarge to read process output line by line. As a small test case I created the attached files, yada.py and yadatest.py. The yada.py script writes its argument repeatedly, with a user-specified delay between each write. Standard output is flushed after each write. Used at the command line, this works as expected.

    The yadatest script exercises yada.py via sarge, capturing stdout and displaying each line as it is received. Again, stdout is flushed after each write.

    The yadatest script runs for a while (about 5 seconds) with no output, and then fails with the following error:

    #!python
    
    >python .\yadatest.py
    Traceback (most recent call last):
      File "yada.py", line 11, in <module>
        print(args.text, flush=True)
    OSError: [Errno 22] Invalid argument
    Exception OSError: OSError(22, 'Invalid argument') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'> ignored
    0 foo
    

    Did something close my child process' stdout? Note that there is no "Error encountered" message, implying that the failure was in the child, not in the parent.

    bug minor 
    opened by vsajip 7
  • ENH: Equivalent to subprocess.check_output

    ENH: Equivalent to subprocess.check_output

    Original report by Wes Turner (Bitbucket: westurner, GitHub: westurner).


    Is there a (easy) way to raise an Exception if the returncode is not (by default?) 0?

    enhancement major 
    opened by vsajip 18
Releases(0.1.7.post1)
Owner
Vinay Sajip
I'm a developer experienced in desktop & Web development. I'm a Python committer - I implemented stdlib logging, venv and the Windows Python launcher.
Vinay Sajip
Rust Markdown Parsing Benchmarks

Rust Markdown Parsing Benchmarks This repo tries to assess Rust markdown parsing

Ed Page 1 Aug 24, 2022
SCTYMN is a GitHub repository that includes some simple scripts(currently only python scripts) that can be useful.

Simple Codes That You Might Need SCTYMN is a GitHub repository that includes some simple scripts(currently only python scripts) that can be useful. In

CodeWriter21 2 Jan 21, 2022
DocumentPy is a Python application that runs in a command-line interface environment, made for creating HTML documents.

DocumentPy DocumentPy is a Python application that runs in a command-line interface environment, made for creating HTML documents. Usage DocumentPy, a

Lotus 0 Jul 15, 2021
Python Deep Dive Course - Accompanying Materials

Python Deep Dive Various Jupyter notebooks and Python sources associated with my Udemy Python 3 Deep Dive course series: Part 1: Mainly functional pro

Fred Baptiste 1.1k Dec 30, 2022
Gaphor is the simple modeling tool

Gaphor Gaphor is a UML and SysML modeling application written in Python. It is designed to be easy to use, while still being powerful. Gaphor implemen

Gaphor 1.3k Jan 03, 2023
A collection of online resources to help you on your Tech journey.

Everything Tech Resources & Projects About The Project Coming from an engineering background and looking to up skill yourself on a new field can be di

Mohamed A 396 Dec 31, 2022
A tutorial for people to run synthetic data replica's from source healthcare datasets

Synthetic-Data-Replica-for-Healthcare Description What is this? A tailored hands-on tutorial showing how to use Python to create synthetic data replic

11 Mar 22, 2022
An ongoing curated list of OS X best applications, libraries, frameworks and tools to help developers set up their macOS Laptop.

macOS Development Setup Welcome to MacOS Local Development & Setup. An ongoing curated list of OS X best applications, libraries, frameworks and tools

Paul Veillard 3 Apr 03, 2022
Plover jyutping - Plover plugin for Jyutping input

Plover plugin for Jyutping Installation Navigate to the repo directory: cd plove

Samuel Lo 1 Mar 17, 2022
An open source utility for creating publication quality LaTex figures generated from OpenFOAM data files.

foamTEX An open source utility for creating publication quality LaTex figures generated from OpenFOAM data files. Explore the docs » Report Bug · Requ

1 Dec 19, 2021
Sphinx-performance - CLI tool to measure the build time of different, free configurable Sphinx-Projects

CLI tool to measure the build time of different, free configurable Sphinx-Projec

useblocks 11 Nov 25, 2022
Data Inspector is an open-source python library that brings 15++ types of different functions to make EDA, data cleaning easier.

Data Inspector Data Inspector is an open-source python library that brings 15 types of different functions to make EDA, data cleaning easier. Author:

Kazi Amit Hasan 38 Nov 24, 2022
Data-Scrapping SEO - the project uses various data scrapping and Google autocompletes API tools to provide relevant points of different keywords so that search engines can be optimized

Data-Scrapping SEO - the project uses various data scrapping and Google autocompletes API tools to provide relevant points of different keywords so that search engines can be optimized; as this infor

Vibhav Kumar Dixit 2 Jul 18, 2022
OpenTelemetry Python API and SDK

Getting Started • API Documentation • Getting In Touch (GitHub Discussions) Contributing • Examples OpenTelemetry Python This page describes the Pytho

OpenTelemetry - CNCF 1.1k Jan 08, 2023
Dynamic Resume Generator

Dynamic Resume Generator

Quinten Lisowe 15 May 19, 2022
A Sublime Text plugin to select a default syntax dialect

Default Syntax Chooser This Sublime Text 4 plugin provides the set_default_syntax_dialect command. This command manipulates a syntax file (e.g.: SQL.s

3 Jan 14, 2022
Swagger UI is a collection of HTML, JavaScript, and CSS assets that dynamically generate beautiful documentation from a Swagger-compliant API.

Introduction Swagger UI allows anyone — be it your development team or your end consumers — to visualize and interact with the API’s resources without

Swagger 23.2k Dec 29, 2022
Gtech μLearn Sample_bot

Ser_bot Gtech μLearn Sample_bot Do Greet a newly joined member in a channel (random message) While adding a reaction to a message send a message to a

Jerin Paul 1 Jan 19, 2022
Speed up Sphinx builds by selectively removing toctrees from some pages

Remove toctrees from Sphinx pages Improve your Sphinx build time by selectively removing TocTree objects from pages. This is useful if your documentat

Executable Books 8 Jan 04, 2023
A fast time mocking alternative to freezegun that wraps libfaketime.

python-libfaketime: fast date/time mocking python-libfaketime is a wrapper of libfaketime for python. Some brief details: Linux and OS X, Pythons 3.5

Simon Weber 68 Jun 10, 2022