Repo for "Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks"

Overview

Summary

This is the code for the paper Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks by Yanxiang Wang, Xian Zhang, Yiran Shen*, Bowen Du, Guangrong Zhao, Lizhen Cui Cui Lizhen, Hongkai Wen.

The paper can be found here.

Introduction

In this paper, We propose new event-based gait recognition approaches basing on two different representations of the event-stream, i.e., graph and image-like representations, and use Graph-based Convolutional Network (GCN) and Convolutional Neural Networks (CNN) respectively to recognize gait from the event-streams. The two approaches are termed as EV-Gait-3DGraph and EV-Gait-IMG. To evaluate the performance of the proposed approaches, we collect two event-based gait datasets, one from real-world experiments and the other by converting the publicly available RGB gait recognition benchmark CASIA-B.

If you use any of this code or data, please cite the following publication:

@inproceedings{wang2019ev,
  title={EV-gait: Event-based robust gait recognition using dynamic vision sensors},
  author={Wang, Yanxiang and Du, Bowen and Shen, Yiran and Wu, Kai and Zhao, Guangrong and Sun, Jianguo and Wen, Hongkai},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={6358--6367},
  year={2019}
}
@article{wang2021event,
 title={Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks},
    author={Wang, Yanxiang and Zhang, Xian and Shen, Yiran and Du, Bowen and Zhao,     Guangrong and Lizhen, Lizhen Cui Cui and Wen, Hongkai},
   journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2021},
   publisher={IEEE}
   }

Requirements

  • Python 3.x
  • Conda
  • cuda
  • PyTorch
  • numpy
  • scipy
  • PyTorch Geometric
  • TensorFlow
  • Matlab (with Computer Vision Toolbox and Image Processing Toolbox for nonuniform grid downsample)

Installation

Data

We use both data collected in real-world experiments(called DVS128-Gait) and converted from publicly available RGB gait databases(called EV-CASIA-B). Here we offer the code and data for the DVS128-Gait.

DVS128-Gait DATASET

we use a DVS128 Dynamic Vision Sensor from iniVation operating at 128*128 pixel resolution.

we collect two dataset: DVS128-Gait-Day and DVS128-Gait-Night, which were collected under day and night lighting condition respectively.

For each lighting condition, we recruited 20 volunteers to contribute their data in two experiment sessions spanning over a few days. In each session, the participants were asked to repeat walking in front of the DVS128 sensor for 100 times.

Run EV-Gait-3DGraph

  • download DVS128-Gait-Day dataset, you will get DVS128-Gait-Day folder which contains train and test data, place DVS128-Gait-Day folder to the data/ folder.

  • event downsample using matlab:

    1. open Matlab
    2. go to matlab_downsample
    3. run main.m. This will generate the data/DVS128-Gait-Day/downsample folder which contains the non-uniform octreeGrid filtering data .
  • or directly download the downsampled data from this link:

    https://pan.baidu.com/s/1OKKvrhid929DakSxsjT7XA , extraction code: ceb1

    Then unzip it to the data/DVS128-Gait-Day/downsample folder.

  • generate graph representation for event, the graph data will be generated in data/DVS128-Gait-Day/graph folder:

    cd generate_graph
    python mat2graph.py
    
  • Download the pretrained model to the trained_model folder:

    https://pan.baidu.com/s/1X7eytUDWAtKS4bk0rjbs6g , extraction code: b7z7

  • run EV-Gait-3DGraph model with the pretrained model:

    cd EV-Gait-3DGraph
    python test_3d_graph.py --model_name EV_Gait_3DGraph.pkl
    

    The parameter--model_name refers to the downloaded pretrained model name.

  • train EV-Gait-3DGraph from scratch:

    cd EV-Gait-3DGraph
    nohup python -u train_3d_graph.py --epoch 110 --cuda 0 > train_3d_graph.log 2>&1 &
    

    the traning log would be created at log/train.log.

    parameters of train_3d_graph.py

    • --batch_size: default 16
    • --epoch: number of iterations, default 150
    • --cuda: specify the cuda device to use, default 0

Run EV-Gait-IMG

  • generate the image-like representation

    cd EV-Gait-IMG
    python make_hdf5.py
    
  • Download the pretrained model to the trained_model folder:

    https://pan.baidu.com/s/1xNbYUYYVPTwwjXeQABjmUw , extraction code: g5k2

    we provide four well trained model for four image-like representations presented in the paper.

    • EV_Gait_IMG_four_channel.pkl
    • EV_Gait_IMG_counts_only_two_channel.pkl
    • EV_Gait_IMG_time_only_two_channel.pkl
    • EV_Gait_IMG_counts_and_time_two_channel.pkl
  • run EV-Gait-IMG model with the pretrained model:

    We provide four options for --img_type to correctly test the corresponding image-like representation

    • four_channel : All four channels are considered, which is the original setup of the image-like representation

      python test_gait_cnn.py --img_type four_channel --model_name EV_Gait_IMG_four_channel.pkl
      
    • counts_only_two_channel : Only the two channels accommodating the counts of positive or negative events are kept

      python test_gait_cnn.py --img_type counts_only_two_channel --model_name EV_Gait_IMG_counts_only_two_channel.pkl
      
    • time_only_two_channel : Only the two channels holding temporal characteristics are kept

      python test_gait_cnn.py --img_type time_only_two_channel --model_name EV_Gait_IMG_time_only_two_channel.pkl
      
    • counts_and_time_two_channel : The polarity of the events is removed

      python test_gait_cnn.py --img_type counts_and_time_two_channel --model_name EV_Gait_IMG_counts_and_time_two_channel.pkl
      

    The parameter --model_name refers to the downloaded pretrained model name.

  • train EV-Gait-IMG from scratch:

    nohup python -u train_gait_cnn.py --img_type counts_only_two_channel --epoch 50 --cuda 1 --batch_size 128 > counts_only_two_channel.log 2>&1 &
    

    parameters of test_gait_cnn.py

    • --batch_size: default 128
    • --epoch: number of iterations, default 50
    • --cuda: specify the cuda device to use, default 0
    • --img_type: specify the type of image-like representation to train the cnn. Four options are provided according to the paper.
      • four_channel : All four channels are considered, which is the original setup of the image-like representation
      • counts_only_two_channel : Only the two channels accommodating the counts of positive or negative events are kept.
      • time_only_two_channel : Only the two channels holding temporal characteristics are kept.
      • counts_and_time_two_channel : The polarity of the events is removed.
Owner
zhangxian
Student
zhangxian
A BaSiC Tool for Background and Shading Correction of Optical Microscopy Images

BaSiC Matlab code accompanying A BaSiC Tool for Background and Shading Correction of Optical Microscopy Images by Tingying Peng, Kurt Thorn, Timm Schr

Marr Lab 34 Dec 18, 2022
Official implementation of the paper Do pedestrians pay attention? Eye contact detection for autonomous driving

Do pedestrians pay attention? Eye contact detection for autonomous driving Official implementation of the paper Do pedestrians pay attention? Eye cont

VITA lab at EPFL 26 Nov 02, 2022
An experimentation and research platform to investigate the interaction of automated agents in an abstract simulated network environments.

CyberBattleSim April 8th, 2021: See the announcement on the Microsoft Security Blog. CyberBattleSim is an experimentation research platform to investi

Microsoft 1.5k Dec 25, 2022
RDA: Robust Domain Adaptation via Fourier Adversarial Attacking

RDA: Robust Domain Adaptation via Fourier Adversarial Attacking Updates 08/2021: check out our domain adaptation for video segmentation paper Domain A

17 Nov 30, 2022
The implementation for "Comprehensive Knowledge Distillation with Causal Intervention".

Comprehensive Knowledge Distillation with Causal Intervention This repository is a PyTorch implementation of "Comprehensive Knowledge Distillation wit

Xiang Deng 10 Nov 03, 2022
Implementation of CVPR 2021 paper "Spatially-invariant Style-codes Controlled Makeup Transfer"

SCGAN Implementation of CVPR 2021 paper "Spatially-invariant Style-codes Controlled Makeup Transfer" Prepare The pre-trained model is avaiable at http

118 Dec 12, 2022
Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation Official PyTorch implementation of the NeurIPS 2021 paper Mingcong Liu, Qiang

onion 462 Dec 29, 2022
This package contains a PyTorch Implementation of IB-GAN of the submitted paper in AAAI 2021

The PyTorch implementation of IB-GAN model of AAAI 2021 This package contains a PyTorch implementation of IB-GAN presented in the submitted paper (IB-

Insu Jeon 9 Mar 30, 2022
PyDeepFakeDet is an integrated and scalable tool for Deepfake detection.

PyDeepFakeDet An integrated and scalable library for Deepfake detection research. Introduction PyDeepFakeDet is an integrated and scalable Deepfake de

Junke, Wang 49 Dec 11, 2022
Denoising Diffusion Implicit Models

Denoising Diffusion Implicit Models (DDIM) Jiaming Song, Chenlin Meng and Stefano Ermon, Stanford Implements sampling from an implicit model that is t

465 Jan 05, 2023
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Shuaifeng Zhi 243 Jan 07, 2023
Generative Exploration and Exploitation - This is an improved version of GENE.

GENE This is an improved version of GENE. In the original version, the states are generated from the decoder of VAE. We have to check whether the gere

33 Mar 23, 2022
PyTorch inference for "Progressive Growing of GANs" with CelebA snapshot

Progressive Growing of GANs inference in PyTorch with CelebA training snapshot Description This is an inference sample written in PyTorch of the origi

320 Nov 21, 2022
Dynamic Attentive Graph Learning for Image Restoration, ICCV2021 [PyTorch Code]

Dynamic Attentive Graph Learning for Image Restoration This repository is for GATIR introduced in the following paper: Chong Mou, Jian Zhang, Zhuoyuan

Jian Zhang 84 Dec 09, 2022
StackNet is a computational, scalable and analytical Meta modelling framework

StackNet This repository contains StackNet Meta modelling methodology (and software) which is part of my work as a PhD Student in the computer science

Marios Michailidis 1.3k Dec 15, 2022
Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance

Models for natural language understanding (NLU) tasks often rely on the idiosyncratic biases of the dataset, which make them brittle against test cases outside the training distribution.

Ubiquitous Knowledge Processing Lab 22 Jan 02, 2023
MarcoPolo is a clustering-free approach to the exploration of bimodally expressed genes along with group information in single-cell RNA-seq data

MarcoPolo is a method to discover differentially expressed genes in single-cell RNA-seq data without depending on prior clustering Overview MarcoPolo

Chanwoo Kim 13 Dec 18, 2022
GLIP: Grounded Language-Image Pre-training

GLIP: Grounded Language-Image Pre-training Updates 12/06/2021: GLIP paper on arxiv https://arxiv.org/abs/2112.03857. Code and Model are under internal

Microsoft 862 Jan 01, 2023
Hardware accelerated, batchable and differentiable optimizers in JAX.

JAXopt Installation | Examples | References Hardware accelerated (GPU/TPU), batchable and differentiable optimizers in JAX. Installation JAXopt can be

Google 621 Jan 08, 2023
Simple reimplemetation experiments about FcaNet

FcaNet-CIFAR An implementation of the paper FcaNet: Frequency Channel Attention Networks on CIFAR10/CIFAR100 dataset. how to run Code: python Cifar.py

76 Feb 04, 2021