PipeLayer is a lightweight Python pipeline framework

Overview

PipeLayer

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications.

Table of Contents

Installation

From the command line:

pip install pipelayer

Getting Started

Step 1: Create Pipeline Filters

hello_world_filters.py

from pipelayer import Filter


class HelloFilter(Filter):
    def run(self, data, context):
        return "Hello"


class WorldFilter(Filter):
    def run(self, data, context):
        return f"{data},  World!"

functions.py

def create_message_dict(data, context):
    return {"message": data}

Step 2: Create a Pipeline

Create a module to run the pipeline:

app.py

from pipelayer import Pipeline

from functions import create_message
from hello_world_filters import HelloFilter, WorldFilter


if __name__ = "__main__":
    hello_world_pipeline = Pipeline([
        HelloFilter,                           # pipeline.Filter type
        WorldFilter,                           # pipeline.Filter instance
        create_message_dict                    # function type
        lambda data, context: json.dumps(data) # anonymous function
    ])

    output = hello_world_pipeline.run()

    # output = '{"message": "Hello, World!"}'

    print(f"Pipeline Output: {output}")
    print(hello_world_pipeline.manifest.__dict__)

Step 3: Run the Pipeline

from the command line:

run app.py

The Framework

pipelayer.Pipeline

__init__(steps, name)

args:

  • steps: List[Union[Step, Callable[[Any, Context], Any]]]
    A list of:

    • Classes and Instances that derive from pipelayer.Filter and implement the run method

    • Classes that implement the pipelayer.Step protocol

    • Functions (instance/class/static/module) that have the following signature

      def func(data: Any, context: Any)
    • Anonymous functions (lambda) with two arguments that follow this pattern:

      my_func = lambda data, context: data
    • Instances of pipelayer.Pipeline

  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

name: str

state: Pipeline.State

steps: List[Union[Step, Callable[[Any, Context], Any]]]

manifest: Manifest
An instance of pipelayer.Manifest that is created when the run method is called.

Methods:

run(data, context) -> Any
The pipeline runner that iterates through the steps and pipes filter output to the next step.

args:

  • data: Any
  • context: pipelayer.Context

pipelayer.Switch

__init__(expression, cases, name)
An implementation of a Switch statement as a pipeline filter

args:

  • expression: Union[Step, Callable[[Any, Context], Any]]
  • cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

expression: Union[Step, Callable[[Any, Context], Any]]
cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
name: Optional[str]
manifest: Manifest

Methods:

run(data, context) -> Any
The switch runner that evaluates the specified expresssion executes the matching case.

pipelayer.Filter

__init__(name, pre_process, post_process)

args:

  • name: Optional[str]
    If not specified, the class name will be used.
  • pre_process: Optional[Callable[[Any, Context], Any]
  • post_process: Optional[Callable[[Any, Context], Any]

Properties:

pre_process: Optional[Callable[[Any, Context], Any]
post_process: Optional[Callable[[Any, Context], Any]

Events:
Events are lists of callables assignable after instantiation and are raised if the pipelayer.filter.raise_events decorator is applied to the implementation of the run method.

start: List[Callable[[Filter, Any], Any]]
Raised before the run method is invoked.

exit: List[Callable[[Filter, Any], Any]]
Raised if action is set to Action.SKIP or Action.EXIT in either a start or stop event handler.

end: List[Callable[[Filter, Any], Any]]
Raised after the run method is invoked.

pipelayer.FilterEventArgs

__init__(data, context, state)

args:

  • data: Any
  • context: Context
  • state: State

pipelayer.Context

A abstract base class for runtime app data.

pipelayer.Manifest

The Manifest keeps a record of Pipeline and Filter activity.

Utilities

pipelayer.util.render_manifest(manifest, indent) -> str
Static function that renders formatted JSON data

args:

  • manifest: Manifest
  • indent: Optional[int]
    Default value is 2.
You might also like...
The Python micro framework for building web applications.

Flask Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to co

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado Web Server Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking ne

Async Python 3.6+ web server/framework | Build fast. Run fast.
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Pyramid - A Python web framework

Pyramid Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more

The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

A familiar HTTP Service Framework for Python.
A familiar HTTP Service Framework for Python.

Responder: a familiar HTTP Service Framework for Python Powered by Starlette. That async declaration is optional. View documentation. This gets you a

The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Comments
  • Python 3.7 - unable to run sample code

    Python 3.7 - unable to run sample code

    Hi,

    With Python 3.7 I cannot run sample code. I have below error:

    Traceback (most recent call last): File "C:/Github/Python/pipelayers/app.py", line 1, in from pipelayer import Pipeline File "C:\Github\Python\venv\lib\site-packages\pipelayer_init_.py", line 6, in from pipelayer.pipeline import Pipeline # NOQA F401 File "C:\Github\Python\venv\lib\site-packages\pipelayer\pipeline.py", line 10, in from pipelayer.protocol import IFilter, IStep File "C:\Github\Python\venv\lib\site-packages\pipelayer\protocol.py", line 5, in from typing import (Any, Callable, List, Optional, Protocol, Tuple, ImportError: cannot import name 'Protocol' from 'typing' (C:\Program Files\Python\lib\typing.py)

    opened by zizu1985 3
  • Version 0.2.0

    Version 0.2.0

    • Adds support for module/static/lambda funcs as pipeline filters and pre/post process functions
    • Adds support type of Filter as a pipeline filter
    • Example project "simple_pipelayer" can run as a standalone app
    • Initializes/validates all filters/processors before running the pipeline
    opened by greater-than 0
Releases(v0.7.0)
Owner
greaterthan
greaterthan
Python implementation of the Javascript Object Signing and Encryption (JOSE) framework

Python implementation of the Javascript Object Signing and Encryption (JOSE) framework

Demonware 94 Nov 20, 2022
Containers And REST APIs Workshop

Containers & REST APIs Workshop Containers vs Virtual Machines Ferramentas Podman: https://podman.io/ Docker: https://www.docker.com/ IBM CLI: https:/

Vanderlei Munhoz 8 Dec 16, 2021
Library for building WebSocket servers and clients in Python

What is websockets? websockets is a library for building WebSocket servers and clients in Python with a focus on correctness and simplicity. Built on

Aymeric Augustin 4.3k Dec 31, 2022
Web3.py plugin for using Flashbots' bundle APIs

This library works by injecting a new module in the Web3.py instance, which allows submitting "bundles" of transactions directly to miners. This is do

Flashbots 293 Dec 31, 2022
A micro web-framework using asyncio coroutines and chained middleware.

Growler master ' dev Growler is a web framework built atop asyncio, the asynchronous library described in PEP 3156 and added to the standard library i

687 Nov 27, 2022
Sierra is a lightweight Python framework for building and integrating web applications

A lightweight Python framework for building and Integrating Web Applications. Sierra is a Python3 library for building and integrating web applications with HTML and CSS using simple enough syntax. Y

83 Sep 23, 2022
Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source

Bionic is fast. It's powered core python without any extra dependencies. Bionic offers stateful hot reload, allowing you to make changes to your code and see the results instantly without restarting

⚓ 0 Mar 05, 2022
The web framework for inventors

Emmett is a full-stack Python web framework designed with simplicity in mind. The aim of Emmett is to be clearly understandable, easy to be learned an

Emmett 796 Dec 26, 2022
TinyAPI - 🔹 A fast & easy and lightweight WSGI Framework for Python

TinyAPI - 🔹 A fast & easy and lightweight WSGI Framework for Python

xArty 3 Apr 08, 2022
News search API developed for the purposes of the ColdCase Project.

Saxion - Cold Case - News Search API Setup Local – Linux/MacOS Make sure you have python 3.9 and pip 21 installed. This project uses a MySQL database,

Dimitar Rangelov 3 Jul 01, 2021
The comprehensive WSGI web application library.

Werkzeug werkzeug German noun: "tool". Etymology: werk ("work"), zeug ("stuff") Werkzeug is a comprehensive WSGI web application library. It began as

The Pallets Projects 6.2k Jan 01, 2023
Dazzler is a Python async UI/Web framework built with aiohttp and react.

Dazzler is a Python async UI/Web framework built with aiohttp and react. Create dazzling fast pages with a layout of Python components and bindings to update from the backend.

Philippe Duval 17 Oct 18, 2022
Low code web framework for real world applications, in Python and Javascript

Full-stack web application framework that uses Python and MariaDB on the server side and a tightly integrated client side library.

Frappe 4.3k Dec 30, 2022
Restful API framework wrapped around MongoEngine

Flask-MongoRest A Restful API framework wrapped around MongoEngine. Setup from flask import Flask from flask_mongoengine import MongoEngine from flask

Close 525 Jan 01, 2023
An easy-to-use high-performance asynchronous web framework.

中文 | English 一个易用的高性能异步 web 框架。 Index.py 文档 Index.py 实现了 ASGI3 接口,并使用 Radix Tree 进行路由查找。是最快的 Python web 框架之一。一切特性都服务于快速开发高性能的 Web 服务。 大量正确的类型注释 灵活且高效的

Index.py 264 Dec 31, 2022
Web framework based on type hint。

Hint API 中文 | English 基于 Type hint 的 Web 框架 hintapi 文档 hintapi 实现了 WSGI 接口,并使用 Radix Tree 进行路由查找。是最快的 Python web 框架之一。一切特性都服务于快速开发高性能的 Web 服务。 大量正确的类型

Aber 19 Dec 02, 2022
Online Boutique is a cloud-native microservices demo application

Online Boutique is a cloud-native microservices demo application. Online Boutique consists of a 10-tier microservices application. The application is

Matt Reider 1 Oct 22, 2021
A comprehensive reference for all topics related to building and maintaining microservices

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find and understand almost anything related to Microservices that i

Ivan Bilan 64 Dec 09, 2022
Flask + Docker + Nginx + Gunicorn + MySQL + Factory Method Pattern

This Flask project is reusable and also an example of how to merge Flask, Docker, Nginx, Gunicorn, MySQL, new: Flask-RESTX, Factory Method design pattern, and other optional dependencies such as Dyna

Facundo Padilla 19 Jul 23, 2022
Free and open source full-stack enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.

Readme web2py is a free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applic

2k Dec 31, 2022