[arXiv] What-If Motion Prediction for Autonomous Driving β“πŸš—πŸ’¨

Related tags

Deep Learningpredictionself-driving-carautonomous-drivingautonomous-vehiclestrajectory-prediction
Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

  1. python >= 3.7
  2. pytorch >= 1.5.0

Installing Dependencies

  1. Install remaining required Python dependencies using pip.

    pip install -r requirements.txt

  2. Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

  1. Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

    β”œβ”€β”€ WIMP
β”‚   β”œβ”€β”€ src
β”‚   β”œβ”€β”€ scripts
β”‚   β”œβ”€β”€ data
β”‚   β”‚   β”œβ”€β”€ argoverse_raw
β”‚   β”‚   β”‚   β”œβ”€β”€ train
β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ *.csv
β”‚   β”‚   β”‚   β”œβ”€β”€ val
β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ *.csv
β”‚   β”‚   β”‚   β”œβ”€β”€ test
β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ *.csv

  2. Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

    python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments
  • Pandas Error runpreprocess.py

    Pandas Error runpreprocess.py

    Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

    opened by ahmetgurhan 0
  • Loss dimensions

    Loss dimensions

    Hi, thank you so much for your fantastic work.

    Which is the order, and the dimensions, in this function?

    def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

    I am not able to obtain good results compared to NLL. I have as inputs:

    predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

    Is this correct?

    opened by Cram3r95 0
  • Reproducing the Map-Free and only Social-Context Results form the Ablation Study

    Reproducing the Map-Free and only Social-Context Results form the Ablation Study

    Hey there,

    I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

    image

    However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

    Looking forward hearing from you soon!

    Best regards

    SchDevel

    opened by SchDevel 2
  • Can I get your inference/visualization code?

    Can I get your inference/visualization code?

    Hi, first of all, thanks for your awesome work and sharing that to us.

    I tried to make inference/visualization code by myself, unfortunately, there were some problems.

    Maybe library's mismatching, my insufficient coding skills, or something else.

    So, can i get your inference/visualization code or even skeleton base code?

    opened by raspbe34 3
  • What is the method for incomplete trajectories?

    What is the method for incomplete trajectories?

    Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

    1. I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete
    2. how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

    Thanks!

    opened by XHwind 0
Releases(1.0)
Owner
William Qi
Prediction @argoai
William Qi
GitHub Repository
Taming Transformers for High-Resolution Image Synthesis

Taming Transformers for High-Resolution Image Synthesis CVPR 2021 (Oral) Taming Transformers for High-Resolution Image Synthesis Patrick Esser*, Robin

CompVis Heidelberg 3.5k Jan 03, 2023
​ This is the Pytorch implementation of Progressive Attentional Manifold Alignment.

PAMA This is the Pytorch implementation of Progressive Attentional Manifold Alignment. Requirements python 3.6 pytorch 1.2.0+ PIL, numpy, matplotlib C

98 Nov 15, 2022
Recursive Bayesian Networks

Recursive Bayesian Networks This repository contains the code to reproduce the results from the NeurIPS 2021 paper Lieck R, Rohrmeier M (2021) Recursi

Robert Lieck 11 Oct 18, 2022
Predicts an answer in yes or no.

Oui-ou-non-prediction Predicts an answer in 'yes' or 'no'. It is based on the game 'effeuiller la marguerite' in which the person plucks flower petals

Ananya Gupta 1 Jan 15, 2022
Suite of 500 procedurally-generated NLP tasks to study language model adaptability

TaskBench500 The TaskBench500 dataset and code for generating tasks. Data The TaskBench dataset is available under wget http://web.mit.edu/bzl/www/Tas

Belinda Li 20 May 17, 2022
Rule Based Classification Project For Python

Rule-Based-Classification-Project (ENG) Business Problem: A game company wants to create new level-based customer definitions (personas) by using some

Deniz Can OĞUZ 4 Oct 29, 2022
Density-aware Single Image De-raining using a Multi-stream Dense Network (CVPR 2018)

DID-MDN Density-aware Single Image De-raining using a Multi-stream Dense Network He Zhang, Vishal M. Patel [Paper Link] (CVPR'18) We present a novel d

He Zhang 224 Dec 12, 2022
Detectron2 is FAIR's next-generation platform for object detection and segmentation.

Detectron2 is Facebook AI Research's next generation software system that implements state-of-the-art object detection algorithms. It is a ground-up r

Facebook Research 23.3k Jan 08, 2023
Unofficial pytorch implementation of 'Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization'

pytorch-AdaIN This is an unofficial pytorch implementation of a paper, Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization [Hua

Naoto Inoue 873 Jan 06, 2023
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models

RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (Paper) (Slides) (Video) RuleBERT is a pre-trained language model that has been fine-tune

16 Aug 24, 2022
CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

UC2 UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu,

Mingyang Zhou 28 Dec 30, 2022
Additional functionality for use with fastai’s medical imaging module

fmi Adding additional functionality to fastai's medical imaging module To learn more about medical imaging using Fastai you can view my blog Install g

14 Oct 31, 2022
QRec: A Python Framework for quick implementation of recommender systems (TensorFlow Based)

Introduction QRec is a Python framework for recommender systems (Supported by Python 3.7.4 and Tensorflow 1.14+) in which a number of influential and

Yu 1.4k Dec 30, 2022
ALBERT-pytorch-implementation - ALBERT pytorch implementation

ALBERT-pytorch-implementation developing... λͺ¨λΈμ˜ κ°œλ…μ΄ν•΄λ₯Ό 돕기 μœ„ν•œ κ΅¬ν˜„λ¬Όλ‘œ ν˜„μž¬ λ³€μˆ˜λͺ…을 μƒμ„Ένžˆ μ μ—ˆκ³ 

BG Kim 3 Oct 06, 2022
DetCo: Unsupervised Contrastive Learning for Object Detection

DetCo: Unsupervised Contrastive Learning for Object Detection arxiv link News Sparse RCNN+DetCo improves from 45.0 AP to 46.5 AP(+1.5) with 3x+ms trai

Enze Xie 234 Dec 18, 2022
IDA file loader for UF2, created for the DEFCON 29 hardware badge

UF2 Loader for IDA The DEFCON 29 badge uses the UF2 bootloader, which conveniently allows you to dump and flash the firmware over USB as a mass storag

Kevin Colley 6 Feb 08, 2022
Lighting the Darkness in the Deep Learning Era: A Survey, An Online Platform, A New Dataset

Lighting the Darkness in the Deep Learning Era: A Survey, An Online Platform, A New Dataset This repository provides a unified online platform, LoLi-P

Chongyi Li 457 Jan 03, 2023
Some pre-commit hooks for OpenMMLab projects

pre-commit-hooks Some pre-commit hooks for OpenMMLab projects. Using pre-commit-hooks with pre-commit Add this to your .pre-commit-config.yaml - rep

OpenMMLab 16 Nov 29, 2022
2021 CCF BDCI ε…¨ε›½δΏ‘ζ―ζ£€η΄’ζŒ‘ζˆ˜ζ―οΌˆCCIR-CupοΌ‰ζ™Ίθƒ½δΊΊζœΊδΊ€δΊ’θ‡ͺη„Άθ―­θ¨€η†θ§£θ΅›ι“η¬¬δΊŒεε‚θ΅›θ§£ε†³ζ–Ήζ‘ˆ

2021 CCF BDCI ε…¨ε›½δΏ‘ζ―ζ£€η΄’ζŒ‘ζˆ˜ζ―(CCIR-Cup) ζ™Ίθƒ½δΊΊζœΊδΊ€δΊ’θ‡ͺη„Άθ―­θ¨€η†θ§£θ΅›ι“η¬¬δΊŒεθ§£ε†³ζ–Ήζ‘ˆ 比衛网址: CCIR-Cup-ζ™Ίθƒ½δΊΊζœΊδΊ€δΊ’θ‡ͺ焢语言理解 1.δΎθ΅–ηŽ―ε’ƒοΌš python==3.8 torch==1.7.1+cu110 numpy==1.19.2 transformers=

JinXiang 22 Oct 29, 2022
[ECE NTUA] πŸ‘ Computer Vision - Lab Projects & Theoretical Problem Sets (2020-2021)

Computer Vision - NTUA (2020-2021) This repository hosts the lab projects and theoretical problem sets of the Computer Vision course held by ECE NTUA

Dimitris Dimos 6 Jul 21, 2022