A torch.Tensor-like DataFrame library supporting multiple execution runtimes and Arrow as a common memory format

Overview

TorchArrow (Warning: Unstable Prototype)

This is a prototype library currently under heavy development. It does not currently have stable releases, and as such will likely be modified significantly in backwards compatibility breaking ways until beta release (targeting early 2022). If you have suggestions on the API or use cases you would like to be covered, please open a GitHub issue. We would love to hear thoughts and feedback.

TorchArrow is a torch.Tensor-like Python DataFrame library for data preprocessing in deep learning. It supports multiple execution runtimes and Arrow as a common format.

It plans to provide:

  • Python Dataframe library implementing streaming-friendly Pandas subset
  • Seamless handoff with PyTorch or other model authoring, such as Tensor collation and easily plugging into PyTorch DataLoader and DataPipes
  • Zero copy for external readers via Arrow in-memory columnar format
  • High-performance CPU backend via Velox
  • GPU backend via libcudf
  • High-performance C++ UDF support with vectorization

Installation

Binaries

Coming soon!

From Source

If you are installing from source, you will need Python 3.8 or later and a C++17 compiler. Also, we highly recommend installing an Miniconda environment.

Get the TorchArrow Source

git clone --recursive https://github.com/facebookresearch/torcharrow
cd torcharrow
# if you are updating an existing checkout
git submodule sync --recursive
git submodule update --init --recursive

Install Dependencies

On MacOS

HomeBrew is required to install development tools on MacOS.

# Install dependencies from Brew
brew install --formula ninja cmake ccache protobuf icu4c boost gflags glog libevent lz4 lzo snappy xz zstd

# Build and install other dependencies
scripts/build_mac_dep.sh ranges_v3 googletest fmt double_conversion folly re2

On Ubuntu (20.04 or later)

# Install dependencies from APT
apt install -y g++ cmake ccache ninja-build checkinstall \
    libssl-dev libboost-all-dev libdouble-conversion-dev libgoogle-glog-dev \
    libbz2-dev libgflags-dev libgtest-dev libgmock-dev libevent-dev libfmt-dev \
    libprotobuf-dev liblz4-dev libzstd-dev libre2-dev libsnappy-dev liblzo2-dev \
    protobuf-compiler
# Build and install Folly
scripts/install_ubuntu_folly.sh

Install TorchArrow

For local development, you can build with debug mode:

DEBUG=1 python setup.py develop

And run unit tests with

python -m unittest -v

To install TorchArrow with release mode (WARNING: may take very long to build):

python setup.py install

Documentation

This 10 minutes tutorial provides a short introduction to TorchArrow. More documents on advanced topics are coming soon!

Future Plans

We hope to sufficiently expand the library, harden APIs, and gather feedback to enable a beta release at the time of the PyTorch 1.11 release (early 2022).

License

TorchArrow is BSD licensed, as found in the LICENSE file.

Comments
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/95b09c7bad6baa93d8f6add4562dfe0cdc8c26cd

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 80
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/0cb50b9fdfccbb277e62e1c2541ae084b29d6080

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 79
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/e432b1df0be62f65e0ba00b4fb966605bcb1443e

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 58
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/7673b382d909add4738240ae0157f2d5cafcf546

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 56
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/5e37e22c974fcd9caceb3dd97a0e84386d188474

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 55
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/08be6833961213b6679a7a7707ca53d486ff84df

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 46
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/41971b30c1cdd9f984018d6a496bc3b99afc7b45

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 36
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/4a36551237993f519dbf5bbae70a4ac9a660bf0d

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 35
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/0e4d3a5efece59b7d6a6a8f23ed7e4668d078e2e

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 33
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/fb7b62bede0beb66cf87fe888d71a9c366fe5ed6

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 33
  • Automated submodule update: velox

    Automated submodule update: velox

    This is an automated pull request to update the first-party submodule for facebookincubator/velox.

    New submodule commit: https://github.com/facebookincubator/velox/commit/b32878fb54eefb01c0c577439d0d6d61644dcff9

    Test Plan: Ensure that CI jobs succeed on GitHub before landing.

    CLA Signed 
    opened by facebook-github-bot 33
  • Stable Release Roadmap

    Stable Release Roadmap

    Hello, I see that the development of the library has slowed down a bit, hence I would like to ask if there exists a roadmap for the first stable release or if there's any other plan for TorchArrow. Thank you very much for your work!

    opened by mbignotti 0
  • `from_arrow` with `List` columns

    `from_arrow` with `List` columns

    Summary: Adds some basic functionality to allow Arrow tables/arrays with List[primitive_type] columns to be converted to a ta.Dataframe.

    Implemented by converting the list column to a pylist and wrapping _from_pysequence. Not super efficient, but provides some functionality to unblock these columns.

    Tests: Modified previous test case that checked for unsupported type. python -m unittest -v

    ----------------------------------------------------------------------
    Ran 196 tests in 1.108s
    
    OK
    
    CLA Signed 
    opened by myzha0 0
  • Generalize Dispatcher class

    Generalize Dispatcher class

    Summary: Generalizing it for reusing it in different contexts. Also changing to global-instance-as-singleton pattern, so that we can instantiate instances for different use cases without worrying about mis-sharing the class variable calls

    Differential Revision: D40188963

    CLA Signed fb-exported 
    opened by OswinC 1
  • Support for arrays in torcharrow.from_arrow

    Support for arrays in torcharrow.from_arrow

    Hi guys! When trying to use ParquetDataFrameLoader I ran across a problem when trying to load parquet file if it has an array field. It looks like it comes down to torcharrow.from_arrow not supporting array columns. But it seems that torcharrow already has support for array columns. Are there any plans to implement it when loading from parquet files or are there any problems which stop this from being implemented?

    The error basically looks like this:

    NotImplementedError                       Traceback (most recent call last)
    Input In [25], in <cell line: 1>()
    ----> 1 next(iter(datapipe))
    
    File /opt/conda/lib/python3.8/site-packages/torch/utils/data/datapipes/_typing.py:514, in hook_iterator.<locals>.wrap_generator(*args, **kwargs)
        512         response = gen.send(None)
        513 else:
    --> 514     response = gen.send(None)
        516 while True:
        517     request = yield response
    
    File /opt/conda/lib/python3.8/site-packages/torch/utils/data/datapipes/iter/combinatorics.py:127, in ShufflerIterDataPipe.__iter__(self)
        125 self._rng.seed(self._seed)
        126 self._seed = None
    --> 127 for x in self.datapipe:
        128     if len(self._buffer) == self.buffer_size:
        129         idx = self._rng.randint(0, len(self._buffer) - 1)
    
    File /opt/conda/lib/python3.8/site-packages/torch/utils/data/datapipes/_typing.py:514, in hook_iterator.<locals>.wrap_generator(*args, **kwargs)
        512         response = gen.send(None)
        513 else:
    --> 514     response = gen.send(None)
        516 while True:
        517     request = yield response
    
    File /opt/conda/lib/python3.8/site-packages/torchdata/datapipes/iter/util/dataframemaker.py:138, in ParquetDFLoaderIterDataPipe.__iter__(self)
        135 for i in range(num_row_groups):
        136     # TODO: More fine-grain control over the number of rows or row group per DataFrame
        137     row_group = parquet_file.read_row_group(i, columns=self.columns, use_threads=self.use_threads)
    --> 138     yield torcharrow.from_arrow(row_group, dtype=self.dtype)
    
    File /opt/conda/lib/python3.8/site-packages/torcharrow/interop.py:32, in from_arrow(data, dtype, device)
         30     return _from_arrow_array(data, dtype, device=device)
         31 elif isinstance(data, pa.Table):
    ---> 32     return _from_arrow_table(data, dtype, device=device)
         33 else:
         34     raise ValueError
    
    File /opt/conda/lib/python3.8/site-packages/torcharrow/interop_arrow.py:86, in _from_arrow_table(table, dtype, device)
         83     field = table.schema.field(i)
         85     assert len(table[i].chunks) == 1
    ---> 86     df_data[field.name] = _from_arrow_array(
         87         table[i].chunk(0),
         88         dtype=(
         89             # pyre-fixme[16]: `DType` has no attribute `get`.
         90             dtype.get(field.name)
         91             if dtype is not None
         92             else _arrowtype_to_dtype(field.type, field.nullable)
         93         ),
         94         device=device,
         95     )
         97 return dataframe(df_data, device=device)
    
    File /opt/conda/lib/python3.8/site-packages/torcharrow/interop_arrow.py:37, in _from_arrow_array(array, dtype, device)
         28 assert isinstance(array, pa.Array)
         30 # Using the most narrow type we can, we (i) don't restrict in any
         31 # way where it can be used (since we can pass a narrower typed
         32 # non-null column to a function expecting a nullable type, but not
       (...)
         35 # increase the amount of places we can use the from_arrow result
         36 # pyre-fixme[16]: `Array` has no attribute `type`.
    ---> 37 dtype_from_arrowtype = _arrowtype_to_dtype(array.type, array.null_count > 0)
         38 if dtype and (
         39     dt.get_underlying_dtype(dtype) != dt.get_underlying_dtype(dtype_from_arrowtype)
         40 ):
         41     raise NotImplementedError("Type casting is not supported")
    
    File /opt/conda/lib/python3.8/site-packages/torcharrow/_interop.py:205, in _arrowtype_to_dtype(t, nullable)
        199 if pa.types.is_struct(t):
        200     return dt.Struct(
        201         # pyre-fixme[16]: `DataType` has no attribute `__iter__`.
        202         [dt.Field(f.name, _arrowtype_to_dtype(f.type, f.nullable)) for f in t],
        203         nullable,
        204     )
    --> 205 raise NotImplementedError(f"Unsupported Arrow type: {str(t)}")
    
    NotImplementedError: Unsupported Arrow type: list<element: float>
    This exception is thrown by __iter__ of ParquetDFLoaderIterDataPipe()
    
    opened by grapefroot 2
Releases(v0.1.0)
  • v0.1.0(Jul 13, 2022)

    We are excited to release the very first Beta version of TorchArrow! TorchArrow is a machine learning preprocessing library over batch data, providing performant and Pandas-style easy-to-use API for model development.

    Highlights

    TorchArrow provides a Python DataFrame that allows extensible UDFs with Velox, with the following features:

    • Seamless handoff with PyTorch or other model authoring, such as Tensor collation and easily plugging into PyTorch DataLoader and DataPipes
    • Zero copy for external readers via Arrow in-memory columnar format
    • Multiple execution runtimes support:
      • High-performance CPU backend via Velox
      • (Future Work) GPU backend via libcudf
    • High-performance C++ UDF support with vectorization

    Installation

    In this release we are supporting install via PYPI: pip install torcharrow.

    Documentation

    You can find the API documentation here.

    This 10 minutes tutorial provides a short introduction to TorchArrow, and you can also try it in this Colab.

    Examples

    You can find the example about integrating a TorchRec based training loop utilizing TorchArrow's on-the-fly preprocessing here. More examples are coming soon!

    Future Plans

    We hope to continue to expand the library, harden API, and gather feedback to enable future releases. Stay tuned!

    Beta Usage Note

    TorchArrow is currently in the Beta stage and does not have a stable release. The API may change based on user feedback or performance. We are committed to bring this library to stable release, but future changes may not be completely backward compatible. If you have suggestions on the API or use cases you'd like to be covered, please open a GitHub issue. We'd love to hear thoughts and feedback.

    Source code(tar.gz)
    Source code(zip)
Owner
Facebook Research
Facebook Research
This is a project based on retinaface face detection, including ghostnet and mobilenetv3

English | 简体中文 RetinaFace in PyTorch Chinese detailed blog:https://zhuanlan.zhihu.com/p/379730820 Face recognition with masks is still robust---------

pogg 59 Dec 21, 2022
Dynamic wallpaper generator.

Wiki • About • Installation About This project is a dynamic wallpaper changer. It waits untill you turn on the music, downloads album cover if it's po

3 Sep 18, 2021
Specification language for generating Generalized Linear Models (with or without mixed effects) from conceptual models

tisane Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships TL;DR: Analysts can use Tisane to author gener

Eunice Jun 11 Nov 15, 2022
Tensors and neural networks in Haskell

Hasktorch Hasktorch is a library for tensors and neural networks in Haskell. It is an independent open source community project which leverages the co

hasktorch 920 Jan 04, 2023
Inteligência artificial criada para realizar interação social com idosos.

IA SONIA 4.0 A SONIA foi inspirada no assistente mais famoso do mundo e muito bem conhecido JARVIS. Todo mundo algum dia ja sonhou em ter o seu própri

Vinícius Azevedo 2 Oct 21, 2021
Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation Efficient Self-Ensemble Framework for Semantic Segmentation by Walid Bousselham

61 Dec 26, 2022
[CVPR 2021] Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach

Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach This is the repo to host the dataset TextSeg and code for TexRNe

SHI Lab 174 Dec 19, 2022
ByteTrack with ReID module following the paradigm of FairMOT, tracking strategy is borrowed from FairMOT/JDE.

ByteTrack_ReID ByteTrack is the SOTA tracker in MOT benchmarks with strong detector YOLOX and a simple association strategy only based on motion infor

Han GuangXin 46 Dec 29, 2022
PyTorch implementation of CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition

PyTorch implementation of CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition The unofficial code of CDistNet. Now, we ha

25 Jul 20, 2022
Implement A3C for Mujoco gym envs

pytorch-a3c-mujoco Disclaimer: my implementation right now is unstable (you ca refer to the learning curve below), I'm not sure if it's my problems. A

Andrew 70 Dec 12, 2022
paper: Hyperspectral Remote Sensing Image Classification Using Deep Convolutional Capsule Network

DC-CapsNet This is a tensorflow and keras based implementation of DC-CapsNet for HSI in the Remote Sensing Letters R. Lei et al., "Hyperspectral Remot

LEI 7 Nov 29, 2022
This is an example implementation of the paper "Cross Domain Robot Imitation with Invariant Representation".

IR-GAIL This is an example implementation of the paper "Cross Domain Robot Imitation with Invariant Representation". Dependency The experiments are de

Zhao-Heng Yin 1 Jul 14, 2022
GE2340 project source code without credentials.

GE2340-Project-Public GE2340 project source code without credentials. Run the bot.py to start the bot Telegram: @jasperwong_ge2340_bot If the bot does

0 Feb 10, 2022
RATE: Overcoming Noise and Sparsity of Textual Features in Real-Time Location Estimation (CIKM'17)

RATE: Overcoming Noise and Sparsity of Textual Features in Real-Time Location Estimation This is the implementation of RATE: Overcoming Noise and Spar

Yu Zhang 5 Feb 10, 2022
Code for "Discovering Non-monotonic Autoregressive Orderings with Variational Inference" (paper and code updated from ICLR 2021)

Discovering Non-monotonic Autoregressive Orderings with Variational Inference Description This package contains the source code implementation of the

Xuanlin (Simon) Li 10 Dec 29, 2022
Lightweight stereo matching network based on MobileNetV1 and MobileNetV2

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Cognitive Systems Research Group 139 Nov 30, 2022
A Moonraker plug-in for real-time compensation of frame thermal expansion

Frame Expansion Compensation A Moonraker plug-in for real-time compensation of frame thermal expansion. Installation Credit to protoloft, from whom I

58 Jan 02, 2023
Code and data of the EMNLP 2021 paper "Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer"

StyleAttack Code and data of the EMNLP 2021 paper "Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer" Prepare Pois

THUNLP 19 Nov 20, 2022
Automatic meme generation model using Tensorflow Keras.

Memefly You can find the project at MemeflyAI. Contributors Nick Buukhalter Harsh Desai Han Lee Project Overview Trello Board Product Canvas Automatic

BloomTech Labs 2 Jan 13, 2022
This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

12 Oct 28, 2022