Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Last update: Nov 22, 2022

Overview

HaloNet - Pytorch

Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This repository will only house the attention layer and not much more.

Install

$ pip install halonet-pytorch

Usage

import torch
from halonet_pytorch import HaloAttention

attn = HaloAttention(
    dim = 512,         # dimension of feature map
    block_size = 8,    # neighborhood block size (feature map must be divisible by this)
    halo_size = 4,     # halo size (block receptive field)
    dim_head = 64,     # dimension of each head
    heads = 4          # number of attention heads
).cuda()

fmap = torch.randn(1, 512, 32, 32).cuda()
attn(fmap) # (1, 512, 32, 32)

Citations

@misc{vaswani2021scaling,
    title   = {Scaling Local Self-Attention For Parameter Efficient Visual Backbones}, 
    author  = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens},
    year    = {2021},
    eprint  = {2103.12731},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

ELSA: Enhanced Local Self-Attention for Vision Transformer By Jingkai Zhou, Pich

87 Dec 19, 2022

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution [arXiv 2021].

122 Dec 12, 2022

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

Focal Transformer This is the official implementation of our Focal Transformer -- "Focal Self-attention for Local-Global Interactions in Vision Transf

486 Dec 20, 2022

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

EfficientNet A PyTorch implementation of EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. [arxiv] [Official TF Repo] Implemen

298 Dec 10, 2022

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

41 Nov 29, 2022

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

YOLOv4-large This is the implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork. YOLOv4-CSP YOLOv4-tiny YOLOv4-

2k Jan 2, 2023

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Relational Self-Attention: What's Missing in Attention for Video Understanding This repository is the official implementation of "Relational Self-Atte

43 Dec 7, 2022

Image-Scaling Attacks and Defenses

Image-Scaling Attacks & Defenses This repository belongs to our publication: Erwin Quiring, David Klein, Daniel Arp, Martin Johns and Konrad Rieck. Ad

163 Nov 21, 2022

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

ImgAlign For auto aligning, cropping, and scaling HR and LR images for training image based neural networks Usage Make sure OpenCV is installed, 'pip

15 Dec 4, 2022

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Related tags

Overview

HaloNet - Pytorch

Install

Usage

Citations

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Image-Scaling Attacks and Defenses

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

Releases(0.0.4)

0.0.4(Mar 24, 2021)

0.0.3(Mar 24, 2021)

0.0.2(Mar 24, 2021)

0.0.1(Mar 24, 2021)

Owner

Phil Wang

ONNX-PackNet-SfM: Python scripts for performing monocular depth estimation using the PackNet-SfM model in ONNX

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Attention for PyTorch with Linear Memory Footprint

Official PyTorch implementation of "Preemptive Image Robustification for Protecting Users against Man-in-the-Middle Adversarial Attacks" (AAAI 2022)

yolov5 deepsort 行人 车辆 跟踪 检测 计数

Official code for paper Exemplar Based 3D Portrait Stylization.

Statistical-Rethinking-with-Python-and-PyMC3 - Python/PyMC3 port of the examples in " Statistical Rethinking A Bayesian Course with Examples in R and Stan" by Richard McElreath

ToFFi - Toolbox for Frequency-based Fingerprinting of Brain Signals

Personals scripts using ageitgey/face_recognition

Dynamical movement primitives (DMPs), probabilistic movement primitives (ProMPs), spatially coupled bimanual DMPs.

This repository contains the exercises and its solution contained in the book "An Introduction to Statistical Learning" in python.

OpenIPDM is a MATLAB open-source platform that stands for infrastructures probabilistic deterioration model

A higher performance pytorch implementation of DeepLab V3 Plus(DeepLab v3+)

Official Chainer implementation of GP-GAN: Towards Realistic High-Resolution Image Blending (ACMMM 2019, oral)

On Generating Extended Summaries of Long Documents

Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering (NAACL 2021)

Conditional Gradients For The Approximately Vanishing Ideal

Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX

PyTorch implementation of paper A Fast Knowledge Distillation Framework for Visual Recognition.

FinEAS: Financial Embedding Analysis of Sentiment 📈

yolov5 deepsort 行人车辆跟踪检测计数