PipeLayer is a lightweight Python pipeline framework

Overview

PipeLayer

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications.

Table of Contents

Installation

From the command line:

pip install pipelayer

Getting Started

Step 1: Create Pipeline Filters

hello_world_filters.py

from pipelayer import Filter


class HelloFilter(Filter):
    def run(self, data, context):
        return "Hello"


class WorldFilter(Filter):
    def run(self, data, context):
        return f"{data},  World!"

functions.py

def create_message_dict(data, context):
    return {"message": data}

Step 2: Create a Pipeline

Create a module to run the pipeline:

app.py

from pipelayer import Pipeline

from functions import create_message
from hello_world_filters import HelloFilter, WorldFilter


if __name__ = "__main__":
    hello_world_pipeline = Pipeline([
        HelloFilter,                           # pipeline.Filter type
        WorldFilter,                           # pipeline.Filter instance
        create_message_dict                    # function type
        lambda data, context: json.dumps(data) # anonymous function
    ])

    output = hello_world_pipeline.run()

    # output = '{"message": "Hello, World!"}'

    print(f"Pipeline Output: {output}")
    print(hello_world_pipeline.manifest.__dict__)

Step 3: Run the Pipeline

from the command line:

run app.py

The Framework

pipelayer.Pipeline

__init__(steps, name)

args:

  • steps: List[Union[Step, Callable[[Any, Context], Any]]]
    A list of:

    • Classes and Instances that derive from pipelayer.Filter and implement the run method

    • Classes that implement the pipelayer.Step protocol

    • Functions (instance/class/static/module) that have the following signature

      def func(data: Any, context: Any)
    • Anonymous functions (lambda) with two arguments that follow this pattern:

      my_func = lambda data, context: data
    • Instances of pipelayer.Pipeline

  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

name: str

state: Pipeline.State

steps: List[Union[Step, Callable[[Any, Context], Any]]]

manifest: Manifest
An instance of pipelayer.Manifest that is created when the run method is called.

Methods:

run(data, context) -> Any
The pipeline runner that iterates through the steps and pipes filter output to the next step.

args:

  • data: Any
  • context: pipelayer.Context

pipelayer.Switch

__init__(expression, cases, name)
An implementation of a Switch statement as a pipeline filter

args:

  • expression: Union[Step, Callable[[Any, Context], Any]]
  • cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

expression: Union[Step, Callable[[Any, Context], Any]]
cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
name: Optional[str]
manifest: Manifest

Methods:

run(data, context) -> Any
The switch runner that evaluates the specified expresssion executes the matching case.

pipelayer.Filter

__init__(name, pre_process, post_process)

args:

  • name: Optional[str]
    If not specified, the class name will be used.
  • pre_process: Optional[Callable[[Any, Context], Any]
  • post_process: Optional[Callable[[Any, Context], Any]

Properties:

pre_process: Optional[Callable[[Any, Context], Any]
post_process: Optional[Callable[[Any, Context], Any]

Events:
Events are lists of callables assignable after instantiation and are raised if the pipelayer.filter.raise_events decorator is applied to the implementation of the run method.

start: List[Callable[[Filter, Any], Any]]
Raised before the run method is invoked.

exit: List[Callable[[Filter, Any], Any]]
Raised if action is set to Action.SKIP or Action.EXIT in either a start or stop event handler.

end: List[Callable[[Filter, Any], Any]]
Raised after the run method is invoked.

pipelayer.FilterEventArgs

__init__(data, context, state)

args:

  • data: Any
  • context: Context
  • state: State

pipelayer.Context

A abstract base class for runtime app data.

pipelayer.Manifest

The Manifest keeps a record of Pipeline and Filter activity.

Utilities

pipelayer.util.render_manifest(manifest, indent) -> str
Static function that renders formatted JSON data

args:

  • manifest: Manifest
  • indent: Optional[int]
    Default value is 2.
You might also like...
The Python micro framework for building web applications.

Flask Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to co

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado Web Server Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking ne

Async Python 3.6+ web server/framework | Build fast. Run fast.
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Pyramid - A Python web framework

Pyramid Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more

The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

A familiar HTTP Service Framework for Python.
A familiar HTTP Service Framework for Python.

Responder: a familiar HTTP Service Framework for Python Powered by Starlette. That async declaration is optional. View documentation. This gets you a

The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Comments
  • Python 3.7 - unable to run sample code

    Python 3.7 - unable to run sample code

    Hi,

    With Python 3.7 I cannot run sample code. I have below error:

    Traceback (most recent call last): File "C:/Github/Python/pipelayers/app.py", line 1, in from pipelayer import Pipeline File "C:\Github\Python\venv\lib\site-packages\pipelayer_init_.py", line 6, in from pipelayer.pipeline import Pipeline # NOQA F401 File "C:\Github\Python\venv\lib\site-packages\pipelayer\pipeline.py", line 10, in from pipelayer.protocol import IFilter, IStep File "C:\Github\Python\venv\lib\site-packages\pipelayer\protocol.py", line 5, in from typing import (Any, Callable, List, Optional, Protocol, Tuple, ImportError: cannot import name 'Protocol' from 'typing' (C:\Program Files\Python\lib\typing.py)

    opened by zizu1985 3
  • Version 0.2.0

    Version 0.2.0

    • Adds support for module/static/lambda funcs as pipeline filters and pre/post process functions
    • Adds support type of Filter as a pipeline filter
    • Example project "simple_pipelayer" can run as a standalone app
    • Initializes/validates all filters/processors before running the pipeline
    opened by greater-than 0
Releases(v0.7.0)
Owner
greaterthan
greaterthan
Containers And REST APIs Workshop

Containers & REST APIs Workshop Containers vs Virtual Machines Ferramentas Podman: https://podman.io/ Docker: https://www.docker.com/ IBM CLI: https:/

Vanderlei Munhoz 8 Dec 16, 2021
Sierra is a lightweight Python framework for building and integrating web applications

A lightweight Python framework for building and Integrating Web Applications. Sierra is a Python3 library for building and integrating web applications with HTML and CSS using simple enough syntax. Y

83 Sep 23, 2022
A framework that let's you compose websites in Python with ease!

Perry Perry = A framework that let's you compose websites in Python with ease! Perry works similar to Qt and Flutter, allowing you to create componen

Linkus 13 Oct 09, 2022
A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs.

cx_Oracle_async A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs. Easy to use , buy may not

36 Dec 21, 2022
REST API framework designed for human beings

Eve Eve is an open source Python REST API framework designed for human beings. It allows to effortlessly build and deploy highly customizable, fully f

eve 6.6k Jan 07, 2023
Sanic integration with Webargs

webargs-sanic Sanic integration with Webargs. Parsing and validating request arguments: headers, arguments, cookies, files, json, etc. IMPORTANT: From

Endurant Devs 13 Aug 31, 2022
web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

5.8k Dec 30, 2022
Djask is a web framework for python which stands on the top of Flask and will be as powerful as Django.

Djask is a web framework for python which stands on the top of Flask and will be as powerful as Django.

Andy Zhou 27 Sep 08, 2022
FPS, fast pluggable server, is a framework designed to compose and run a web-server based on plugins.

FPS, fast pluggable server, is a framework designed to compose and run a web-server based on plugins. It is based on top of fastAPI, uvicorn, typer, and pluggy.

Adrien Delsalle 1 Nov 16, 2021
The web framework for inventors

Emmett is a full-stack Python web framework designed with simplicity in mind. The aim of Emmett is to be clearly understandable, easy to be learned an

Emmett 796 Dec 26, 2022
Fast, asynchronous and elegant Python web framework.

Warning: This project is being completely re-written. If you're curious about the progress, reach me on Slack. Vibora is a fast, asynchronous and eleg

vibora.io 5.7k Jan 08, 2023
Bromelia-hss implements an HSS by using the Python micro framework Bromélia.

Bromélia HSS bromelia-hss is the second official implementation of a Diameter-based protocol application by using the Python micro framework Bromélia.

henriquemr 7 Nov 02, 2022
easyopt is a super simple yet super powerful optuna-based Hyperparameters Optimization Framework that requires no coding.

easyopt is a super simple yet super powerful optuna-based Hyperparameters Optimization Framework that requires no coding.

Federico Galatolo 9 Feb 04, 2022
Python AsyncIO data API to manage billions of resources

Introduction Please read the detailed docs This is the working project of the next generation Guillotina server based on asyncio. Dependencies Python

Plone Foundation 183 Nov 15, 2022
A library that makes consuming a RESTful API easier and more convenient

Slumber is a Python library that provides a convenient yet powerful object-oriented interface to ReSTful APIs. It acts as a wrapper around the excellent requests library and abstracts away the handli

Sam Giles 597 Dec 13, 2022
A beginners course for Django

The Definitive Django Learning Platform. Getting started with Django This is the code from the course "Getting Started With Django", found on YouTube

JustDjango 288 Jan 08, 2023
A high-level framework for building GitHub applications in Python.

A high-level framework for building GitHub applications in Python. Core Features Async Proper ratelimit handling Handles interactions for you (

Vish M 3 Apr 12, 2022
Python Wrapper for interacting with the Flutterwave API

Python Flutterwave Description Python Wrapper for interacting with the Flutterwa

William Otieno 32 Dec 14, 2022
A comprehensive reference for all topics related to building and maintaining microservices

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find and understand almost anything related to Microservices that i

Ivan Bilan 64 Dec 09, 2022
Serverless Python

Zappa - Serverless Python About Installation and Configuration Running the Initial Setup / Settings Basic Usage Initial Deployments Updates Rollback S

Rich Jones 11.9k Jan 01, 2023