poetry2nix turns Poetry projects into Nix derivations without the need to actually write Nix expressions

Overview

poetry2nix

poetry2nix turns Poetry projects into Nix derivations without the need to actually write Nix expressions. It does so by parsing pyproject.toml and poetry.lock and converting them to Nix derivations on the fly.

For more information, see the announcement post on the Tweag blog. Development of poetry2nix has been supported by Tweag.

API

The poetry2nix public API consists of the following attributes:

  • mkPoetryApplication: Creates a Python application.
  • mkPoetryEnv: Creates a Python environment with an interpreter and all packages from poetry.lock.
  • mkPoetryPackages: Creates an attribute set providing access to the generated packages and other artifacts.
  • mkPoetryScriptsPackage: Creates a package containing the scripts from tool.poetry.scripts of the pyproject.toml.
  • mkPoetryEditablePackage: Creates a package containing editable sources. Changes in the specified paths will be reflected in an interactive nix-shell session without the need to restart it.
  • defaultPoetryOverrides: A set of bundled overrides fixing problems with Python packages.
  • overrides.withDefaults: A convenience function for specifying overrides on top of the defaults.
  • overrides.withoutDefaults: A convenience function for specifying overrides without defaults.
  • cleanPythonSources: A function to create a source filter for python projects.

mkPoetryApplication

Creates a Python application using the Python interpreter specified based on the designated poetry project and lock files. mkPoetryApplication takes an attribute set with the following attributes (attributes without default are mandatory):

  • projectDir: path to the root of the project.
  • src: project source (default: cleanPythonSources { src = projectDir; }).
  • pyproject: path to pyproject.toml (default: projectDir + "/pyproject.toml").
  • poetrylock: poetry.lock file path (default: projectDir + "/poetry.lock").
  • overrides: Python overrides to apply (default: [defaultPoetryOverrides]).
  • meta: application meta data (default: {}).
  • python: The Python interpreter to use (default: pkgs.python3).

Other attributes are passed through to buildPythonApplication.

Example

poetry2nix.mkPoetryApplication {
    projectDir = ./.;
}

See ./pkgs/poetry/default.nix for a working example.

Dependency environment

The resulting derivation also has the passthru attribute dependencyEnv, which is an environment with a python interpreter, all non-development dependencies and your application. This can be used if your application doesn't provide any binaries on its own and instead relies on dependencies binaries to call its modules (as is often the case with celery or gunicorn). For example, if your application defines a CLI for the module admin and a gunicorn app for the module web, a working default.nix would contain

let
    app = poetry2nix.mkPoetryApplication {
        projectDir = ./.;
    };
in app.dependencyEnv

After building this expression, your CLI and app can be called with these commands

$ result/bin/python -m admin
$ result/bin/gunicorn web:app

Note: If you need to perform overrides on the application, use app.dependencyEnv.override { app = app.override { ... }; }. See ./tests/dependency-environment/default.nix for a full example.

mkPoetryEnv

Creates an environment that provides a Python interpreter along with all dependencies declared by the designated poetry project and lock files. Also allows package sources of an application to be installed in editable mode for fast development. mkPoetryEnv takes an attribute set with the following attributes (attributes without default are mandatory):

  • projectDir: path to the root of the project.
  • pyproject: path to pyproject.toml (default: projectDir + "/pyproject.toml").
  • poetrylock: poetry.lock file path (default: projectDir + "/poetry.lock").
  • overrides: Python overrides to apply (default: [defaultPoetryOverrides]).
  • python: The Python interpreter to use (default: pkgs.python3).
  • editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode (default: {}).

Example

poetry2nix.mkPoetryEnv {
    projectDir = ./.;
}

See ./tests/env/default.nix for a working example.

Example with editable packages

poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    editablePackageSources = {
        my-app = ./src;
    };
}

See ./tests/editable/default.nix for a working example of an editable package.

Example shell.nix

The env attribute of the attribute set created by mkPoetryEnv contains a shell environment.

{ pkgs ? import 
    {} }:
let
  myAppEnv = pkgs.poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    editablePackageSources = {
      my-app = ./src;
    };
  };
in myAppEnv.env

Example shell.nix with external dependencies

For a shell environment including external dependencies, override the env to add dependency packages (for example, pkgs.hello) as build inputs.

{ pkgs ? import 
    {} }:
let
  myAppEnv = pkgs.poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    editablePackageSources = {
      my-app = ./src;
    };
  };
in myAppEnv.env.overrideAttrs (oldAttrs: {
  buildInputs = [ pkgs.hello ]:
})

mkPoetryPackages

Creates an attribute set of the shape { python, poetryPackages, pyProject, poetryLock }. Where python is the interpreter specified, poetryPackages is a list of all generated python packages, pyProject is the parsed pyproject.toml and poetryLock is the parsed poetry.lock file. mkPoetryPackages takes an attribute set with the following attributes (attributes without default are mandatory):

  • projectDir: path to the root of the project.
  • pyproject: path to pyproject.toml (default: projectDir + "/pyproject.toml").
  • poetrylock: poetry.lock file path (default: projectDir + "/poetry.lock").
  • overrides: Python overrides to apply (default: [defaultPoetryOverrides]).
  • python: The Python interpreter to use (default: pkgs.python3).
  • editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode (default: {}).

Example

poetry2nix.mkPoetryPackages {
    projectDir = ./.;
    python = python35;
}

mkPoetryScriptsPackage

Creates a package containing the scripts from tool.poetry.scripts of the pyproject.toml:

  • projectDir: path to the root of the project.
  • pyproject: path to pyproject.toml (default: projectDir + "/pyproject.toml").
  • python: The Python interpreter to use (default: pkgs.python3).

Example

poetry2nix.mkPoetryScriptsPackage {
    projectDir = ./.;
    python = python35;
}

mkPoetryEditablePackage

Creates a package containing editable sources. Changes in the specified paths will be reflected in an interactive nix-shell session without the need to restart it:

  • projectDir: path to the root of the project.
  • pyproject: path to pyproject.toml (default: projectDir + "/pyproject.toml").
  • python: The Python interpreter to use (default: pkgs.python3).
  • editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode (default: {}).

Example

poetry2nix.mkPoetryEditablePackage {
    projectDir = ./.;
    python = python35;
    editablePackageSources = {
        my-app = ./src;
    };
}

defaultPoetryOverrides

poetry2nix bundles a set of default overrides that fix problems with various Python packages. These overrides are implemented in overrides.nix.

overrides.withDefaults

Returns a list containing the specified overlay and defaultPoetryOverrides.

Takes an attribute set with the following attributes (attributes without default are mandatory):

  • src: project source directory

Example

poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    overrides = poetry2nix.overrides.withDefaults (self: super: { foo = null; });
}

See ./tests/override-support/default.nix for a working example.

overrides.withoutDefaults

Returns a list containing just the specified overlay, ignoring defaultPoetryOverrides.

Example

poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    overrides = poetry2nix.overrides.withoutDefaults (self: super: { foo = null; });
}

cleanPythonSources

Provides a source filtering mechanism that:

  • Filters gitignore's
  • Filters pycache/pyc files
  • Uses cleanSourceFilter to filter out .git/.hg, .o/.so, editor backup files & nix result symlinks

Example

poetry2nix.cleanPythonSources {
    src = ./.;
}

Creating a custom Poetry2nix instance

Sometimes when it can be convenient to create a custom instance of poetry2nix with a different set of default overrides.

Example

let
  # self & super refers to poetry2nix
  p2nix = poetry2nix.overrideScope' (self: super: {

    # pyself & pysuper refers to python packages
    defaultPoetryOverrides = super.defaultPoetryOverrides.extend (pyself: pysuper: {

      my-custom-pkg = super.my-custom-pkg.overridePythonAttrs (oldAttrs: { });

    });

  });

in
p2nix.mkPoetryApplication {
  projectDir = ./.;
}

or as a nixpkgs overlay:

let
  pkgs = import 
    {
    overlays = [
      # self & super refers to nixpkgs
      (self: super: {

        # p2self & p2super refers to poetry2nix
        poetry2nix = super.poetry2nix.overrideScope' (p2nixself: p2nixsuper: {

          # pyself & pysuper refers to python packages
          defaultPoetryOverrides = p2nixsuper.defaultPoetryOverrides.extend (pyself: pysuper: {

            my-custom-pkg = super.my-custom-pkg.overridePythonAttrs (oldAttrs: { });

          });

        });
      })

    ];
  };

in pkgs.poetry2nix.mkPoetryApplication {
  projectDir = ./.;
}

Using the flake

For the experimental flakes functionality we provide poetry2nix as a flake providing an overlay to use with nixpkgs. Additionally, the flake provides a flake template to quickly start using poetry2nix in a project:

nix flake init --template github:nix-community/poetry2nix

Contributing

Contributions to this project are welcome in the form of GitHub PRs. Please consider the following before creating PRs:

  • This project uses nixpkgs-fmt for formatting the Nix code. You can use nix-shell --run "nixpkgs-fmt ." to format everything.
  • If you are planning to make any considerable changes, you should first present your plans in a GitHub issue so it can be discussed.
  • If you add new features please consider adding tests. You can run them locally as follows:
nix-build --keep-going --show-trace tests/default.nix

License

poetry2nix is released under the terms of the MIT license.

Owner
Nix community projects
A project incubator that works in parallel of the @NixOS org
Nix community projects
Simulation-Based Inference Benchmark

This repository contains a simulation-based inference benchmark framework, sbibm, which we describe in the associated manuscript "Benchmarking Simulation-based Inference".

SBI Benchmark 58 Oct 13, 2022
Extrator de dados do jupiterweb

Extrator de dados do jupiterweb O programa é composto de dois arquivos: Um constando apenas de classes complementares que representam as unidades e as

Bruno Aricó 2 Nov 28, 2022
Simple package to make requests throughout Tor with circuit renewal.

AutoTor Table of Contents About the Project Contents Dependencies Getting Started Installation Coding Contributing About the Project Simple package to

Salvador Belenguer 6 Jan 01, 2023
A python script for compiling and executing .cc files

Debug And Run A python script for compiling and executing .cc files Example dbrun fname.cc [DEBUG MODE] Compiling fname.cc with C++17 ------------

1 May 28, 2022
Retrieve bank transactions and categorize for budgeting use

Budgeting After trying out some budgeting software, I decided to make my own. selenium_scraper Using the selenium package, this script runs an instanc

Marc 1 Nov 10, 2021
Swim between bookmarks in the Windows terminal

Marlin Swim between bookmarks in the terminal! Marlin is an easy to use bookmark manager for the terminal. Choose a folder, bookmark it and swim there

wilfredinni 7 Nov 03, 2022
A multi-platform fuzzer for poking at userland binaries and servers

litefuzz A multi-platform fuzzer for poking at userland binaries and servers litefuzz intro why how it works what it does what it doesn't do support p

52 Nov 18, 2022
Osintgram by Datalux but i fixed some errors i found and made it look cleaner

OSINTgram-V2 OSINTgram-V2 is made from Osintgram which is made by Datalux originally but i took the script and fixed some errors i found and made the

2 Feb 02, 2022
Demo of using DataLoader to prevent out of memory

Demo of using DataLoader to prevent out of memory

3 Jun 25, 2022
A simple python script where the user inputs the current ingredients they have in their kitchen into ingredients.txt

A simple python script where the user inputs the current ingredients they have in their kitchen into ingredients.txt and then runs the main.py script, and it will output what recipes can be created b

Jordan Leich 3 Nov 02, 2022
Fisherman is a free open source fishing bot written in python.

Fisherman is a free open source fishing bot written in python.

Pure | Cody 33 Jan 29, 2022
This is collection of Managementsystem programs: Hospital Management, Student Managemen, etc

Contribute in this repository and help other students with their assignment by adding python scripts for various management system programs.

GDSC BVP DET - Navi Mumbai 3 Mar 20, 2022
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

MetaCall 10 Dec 28, 2022
ARK sõidueksami Matrixi bot

ARK Sõidueksami bot Küsib ARK-i lehelt uusimad eksami ajad ja saadab sõnumi Matrixi kanali Dev setup Linux python3 -m venv venv source venv/bin/activa

Arti Zirk 3 Jun 15, 2021
Model Quantization Benchmark

MQBench Update V0.0.2 Fix academic prepare setting. More deployable prepare process. Fix setup.py. Fix deploy on SNPE. Fix convert_deploy bug. Add Qua

500 Jan 06, 2023
A python program for rick rolling people.

Rickware A python program for rick rolling people. (And annoying them too) What is rick roll? Read this wikipedia article - Rickrolling About program

2 Jan 18, 2022
Transform your boring distro into a hacking powerhouse.

Pentizer Transform your boring distro into a hacking powerhouse. Pentizer is a personal project that imports Kali and Parrot repositories in any Debia

Michail Tsimpliarakis 2 Nov 05, 2021
Job Guy Backend

جاب‌گای چیست؟ اونجا وضعیت چطوریه؟ یه سوال به همین کلیت و ابهام معمولا وقتی برای یه شرکت رزومه می‌فرستیم این سوال کلی و بزرگ برای همه پیش میاد.اونجا وض

Jobguy.work 217 Dec 25, 2022
A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

Sebastián A. Aedo Quililongo 9 Nov 18, 2022
Blender 3.1 Alpha (and later) PLY importer that correctly loads point clouds (and all PLY models as point clouds)

import-ply-as-verts Blender 3.1 Alpha (and later) PLY importer that correctly loads point clouds (and all PLY models as point clouds) Latest News Mand

Michael Prostka 82 Dec 20, 2022