The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Last update: Oct 19, 2022

Overview

Enformer TPU training script (wip)

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pytorch.

This was pieced together from the Deepmind Enformer repository, the colab training notebook, as well as Basenji sequence augmentation code

It accounts for:

distributed TPU training
distributed datasets
distributed validation
gradient clipping
cross replica batchnorms
dataset augmentation

Training takes about 3 days on v3-64

Todo

fix script for differences in sequence length in basenji training data, which is ~130k vs ~190k bp as in paper

Citations

@article {Avsec2021.04.07.438649,
    author  = {Avsec, {\v Z}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},
    title   = {Effective gene expression prediction from sequence by integrating long-range interactions},
    elocation-id = {2021.04.07.438649},
    year    = {2021},
    doi     = {10.1101/2021.04.07.438649},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649},
    eprint  = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649.full.pdf},
    journal = {bioRxiv}
}

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Related tags

Overview

Enformer TPU training script (wip)

Todo

Citations

Owner

Phil Wang

Latex code for making neural networks diagrams

Unbalanced Feature Transport for Exemplar-based Image Translation (CVPR 2021)

Build an Amazon SageMaker Pipeline to Transform Raw Texts to A Knowledge Graph

Repository for the paper "Exploring the Sensory Spaces of English Perceptual Verbs in Natural Language Data"

TensorFlow GNN is a library to build Graph Neural Networks on the TensorFlow platform.

pyspark🍒🥭 is delicious，just eat it!😋😋

PyTorch implementation of probabilistic deep forecast applied to air quality.

O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis

Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

A Pytorch implementation of the multi agent deep deterministic policy gradients (MADDPG) algorithm

Benchmark library for high-dimensional HPO of black-box models based on Weighted Lasso regression

bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED)

A simple Python library for stochastic graphical ecological models

WRENCH: Weak supeRvision bENCHmark

Learning RAW-to-sRGB Mappings with Inaccurately Aligned Supervision (ICCV 2021)

Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

Hidden-Fold Networks (HFN): Random Recurrent Residuals Using Sparse Supermasks

This is a model made out of Neural Network specifically a Convolutional Neural Network model

A pure PyTorch batched computation implementation of "CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition"

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.