Repo for "Physion: Evaluating Physical Prediction from Vision in Humans and Machines" submission to NeurIPS 2021 (Datasets & Benchmarks track)

Overview

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Animation of the 8 scenarios

This repo contains code and data to reproduce the results in our paper, Physion: Evaluating Physical Prediction from Vision in Humans and Machines. Please see below for details about how to download the Physion dataset, replicate our modeling & human experiments, and statistical analyses to reproduce our results.

  1. Downloading the Physion dataset
  2. Dataset generation
  3. Modeling experiments
  4. Human experiments
  5. Comparing models and humans

Downloading the Physion dataset

Downloading the Physion test set (a.k.a. stimuli)

PhysionTest-Core (270 MB)

PhysionTest-Core is all you need to evaluate humans and models on exactly the same test stimuli used in our paper.

It contains eight directories, one for each scenario type (e.g., collide, contain, dominoes, drape, drop, link, roll, support).

Each of these directories contains three subdirectories:

  • maps: Contains PNG segmentation maps for each test stimulus, indicating location of agent object in red and patient object in yellow.
  • mp4s: Contains the MP4 video files presented to human participants. The agent and patient objects appear in random colors.
  • mp4s-redyellow: Contains the MP4 video files passed into models. The agent and patient objects consistently appear in red and yellow, respectively.

Download URL: https://physics-benchmarking-neurips2021-dataset.s3.amazonaws.com/Physion.zip.

PhysionTest-Complete (380 GB)

PhysionTest-Complete is what you want if you need more detailed metadata for each test stimulus.

Each stimulus is encoded in an HDF5 file containing comprehensive information regarding depth, surface normals, optical flow, and segmentation maps associated with each frame of each trial, as well as other information about the physical states of objects at each time step.

Download URL: https://physics-benchmarking-neurips2021-dataset.s3.amazonaws.com/PhysionTestHDF5.tar.gz.

You can also download the testing data for individual scenarios from the table in the next section.

Downloading the Physion training set

Downloading PhysionTrain-Dynamics

PhysionTrain-Dynamics contains the full dataset used to train the dynamics module of models benchmarked in our paper. It consists of approximately 2K stimuli per scenario type.

Download URL (770 MB): https://physics-benchmarking-neurips2021-dataset.s3.amazonaws.com/PhysionTrainMP4s.tar.gz

Downloading PhysionTrain-Readout

PhysionTrain-Readout contains a separate dataset used for training the object-contact prediction (OCP) module for models pretrained on the PhysionTrain-Dynamics dataset. It consists of 1K stimuli per scenario type.

The agent and patient objects in each of these readout stimuli consistently appear in red and yellow, respectively (as in the mp4s-redyellow examples from PhysionTest-Core above).

NB: Code for using these readout sets to benchmark any pretrained model (not just models trained on the Physion training sets) will be released prior to publication.

Download URLs for complete PhysionTrain-Dynamics and PhysionTrain-Readout:

Scenario Dynamics Training Set Readout Training Set Test Set
Dominoes Dominoes_dynamics_training_HDF5s Dominoes_readout_training_HDF5s Dominoes_testing_HDF5s
Support Support_dynamics_training_HDF5s Support_readout_training_HDF5s Support_testing_HDF5s
Collide Collide_dynamics_training_HDF5s Collide_readout_training_HDF5s Collide_testing_HDF5s
Contain Contain_dynamics_training_HDF5s Contain_readout_training_HDF5s Contain_testing_HDF5s
Drop Drop_dynamics_training_HDF5s Drop_readout_training_HDF5s Drop_testing_HDF5s
Roll Roll_dynamics_training_HDF5s Roll_readout_training_HDF5s Roll_testing_HDF5s
Link Link_dynamics_training_HDF5s Link_readout_training_HDF5s Link_testing_HDF5s
Drape Drape_dynamics_training_HDF5s Drape_readout_training_HDF5s Drape_testing_HDF5s

Dataset generation

This repo depends on outputs from tdw_physics.

Specifically, tdw_physics is used to generate the dataset of physical scenarios (a.k.a. stimuli), including both the training datasets used to train physical-prediction models, as well as test datasets used to measure prediction accuracy in both physical-prediction models and human participants.

Instructions for using the ThreeDWorld simulator to regenerate datasets used in our work can be found here. Links for downloading the Physion testing, training, and readout fitting datasets can be found here.

Modeling experiments

The modeling component of this repo depends on the physopt repo. The physopt repo implements an interface through which a wide variety of physics prediction models from the literature (be they neural networks or otherwise) can be adapted to accept the inputs provided by our training and testing datasets and produce outputs for comparison with our human measurements.

The physopt also contains code for model training and evaluation. Specifically, physopt implements three train/test procols:

  • The only protocol, in which each candidate physics model architecture is trained -- using that model's native loss function as specified by the model's authors -- separately on each of the scenarios listed above (e.g. "dominoes", "support", &c). This produces eight separately-trained models per candidate architecture (one for each scenario). Each of these separate models are then tested in comparison to humans on the testing data for that scenario.
  • A all protocol, in which each candidate physics architecture is trained on mixed data from all of the scenarios simultaneously (again, using that model's native loss function). This single model is then tested and compared to humans separately on each scenario.
  • A all-but-one protocol, in which each candidate physics architecture is trained on mixed data drawn for all but one scenario -- separately for all possible choices of the held-out scenario. This produces eight separately-trained models per candidate architecture (one for each held-out scenario). Each of these separate models are then tested in comparison to humans on the testing data for that scenario.

Results from each of the three protocols are separately compared to humans (as described below in the section on comparison of humans to models). All model-human comparisons are carried using a representation-learning paradigm, in which models are trained on their native loss functions (as encoded by the original authors of the model). Trained models are then evaluated on the specific physion red-object-contacts-yellow-zone prediction task. This evaluation is carried by further training a "readout", implemented as a linear logistic regression. Readouts are always trained in a per-scenario fashion.

Currently, physopt implements the following specific physics prediction models:

Model Name Our Code Link Original Paper Description
SVG Denton and Fergus 2018 Image-like latent
OP3 Veerapaneni et. al. 2020
CSWM Kipf et. al. 2020
RPIN Qi et. al. 2021
pVGG-mlp
pVGG-lstm
pDEIT-mlp Touvron et. al. 2020
pDEIT-lstm
GNS Sanchez-Gonzalez et. al. 2020
GNS-R
DPI Li et. al. 2019

Human experiments

This repo contains code to conduct the human behavioral experiments reported in this paper, as well as analyze the resulting data from both human and modeling experiments.

The details of the experimental design and analysis plan are documented in our study preregistration contained within this repository. The format for this preregistration is adapted from the templates provided by the Open Science Framework for our studies, and put under the same type of version control as the rest of the codebase for this project.

Here is what each main directory in this repo contains:

  • experiments: This directory contains code to run the online human behavioral experiments reported in this paper. More detailed documentation of this code can be found in the README file nested within the experiments subdirectory.
  • analysis (aka notebooks): This directory contains our analysis jupyter/Rmd notebooks. This repo assumes you have also imported model evaluation results from physopt.
  • results: This directory contains "intermediate" results of modeling/human experiments. It contains three subdirectories: csv, plots, and summary.
    • /results/csv/ contains csv files containing tidy dataframes with "raw" data.
    • /results/plots/ contains .pdf/.png plots, a selection of which are then polished and formatted for inclusion in the paper using Adobe Illustrator.
    • Important: Before pushing any csv files containing human behavioral data to a public code repository, triple check that this data is properly anonymized. This means no bare AMT Worker ID's or Prolific participant IDs.
  • stimuli: This directory contains any download/preprocessing scripts for data (a.k.a. stimuli) that are the inputs to human behavioral experiments. This repo assumes you have generated stimuli using tdw_physics. This repo uses code in this directory to upload stimuli to AWS S3 and generate metadata to control the timeline of stimulus presentation in the human behavioral experiments.
  • utils: This directory is meant to contain any files containing general helper functions.

Comparing models and humans

The results reported in this paper can be reproduced by running the Jupyter notebooks contained in the analysis directory.

  1. Downloading results. To download the "raw" human and model prediction behavior, please navigate to the analysis directory and execute the following command at the command line: python download_results.py. This script will fetch several CSV files and download them to subdirectories within results/csv. If this does not work, please download this zipped folder (csv) and move it to the results directory: https://physics-benchmarking-neurips2021-dataset.s3.amazonaws.com/model_human_results.zip.
  2. Reproducing analyses. To reproduce the key analyses reported in the paper, please run the following notebooks in this sequence:
    • summarize_human_model_behavior.ipynb: The purpose of this notebook is to:
      • Apply preprocessing to human behavioral data
      • Visualize distribution and compute summary statistics over human physical judgments
      • Visualize distribution and compute summary statistics over model physical judgments
      • Conduct human-model comparisons
      • Output summary CSVs that can be used for further statistical modeling & create publication-quality visualizations
    • inference_human_model_behavior.ipynb: The purpose of this notebook is to:
      • Visualize human and model prediction accuracy (proportion correct)
      • Visualize average-human and model agreement (RMSE)
      • Visualize human-human and model-human agreement (Cohen's kappa)
      • Compare performance between models
    • paper_plots.ipynb: The purpose of this notebook is to create publication-quality figures for inclusion in the paper.
Owner
Cognitive Tools Lab
reverse engineering the human cognitive toolkit
Cognitive Tools Lab
Python implementation of MULTIseq barcode alignment using fuzzy string matching and GMM barcode assignment

Python implementation of MULTIseq barcode alignment using fuzzy string matching and GMM barcode assignment.

MT Schmitz 2 Feb 11, 2022
CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability

This is the official repository of the paper: CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability A private copy of the

Fadi Boutros 33 Dec 31, 2022
Sentiment analysis translations of the Bhagavad Gita

Sentiment and Semantic Analysis of Bhagavad Gita Translations It is well known that translations of songs and poems not only breaks rhythm and rhyming

Machine learning and Bayesian inference @ UNSW Sydney 3 Aug 01, 2022
2021-MICCAI-Progressively Normalized Self-Attention Network for Video Polyp Segmentation

2021-MICCAI-Progressively Normalized Self-Attention Network for Video Polyp Segmentation Authors: Ge-Peng Ji*, Yu-Cheng Chou*, Deng-Ping Fan, Geng Che

Ge-Peng Ji (Daniel) 85 Dec 30, 2022
A library for building and serving multi-node distributed faiss indices.

About Distributed faiss index service. A lightweight library that lets you work with FAISS indexes which don't fit into a single server memory. It fol

Meta Research 170 Dec 30, 2022
Official implementation for paper: A Latent Transformer for Disentangled Face Editing in Images and Videos.

A Latent Transformer for Disentangled Face Editing in Images and Videos Official implementation for paper: A Latent Transformer for Disentangled Face

InterDigital 108 Dec 09, 2022
MODNet: Trimap-Free Portrait Matting in Real Time

MODNet is a model for real-time portrait matting with only RGB image input.

Zhanghan Ke 2.8k Dec 30, 2022
Patch-Based Deep Autoencoder for Point Cloud Geometry Compression

Patch-Based Deep Autoencoder for Point Cloud Geometry Compression Overview The ever-increasing 3D application makes the point cloud compression unprec

17 Dec 05, 2022
Time Delayed NN implemented in pytorch

Pytorch Time Delayed NN Time Delayed NN implemented in PyTorch. Usage kernels = [(1, 25), (2, 50), (3, 75), (4, 100), (5, 125), (6, 150)] tdnn = TDNN

Daniil Gavrilov 79 Aug 04, 2022
PyTorch Implementation for AAAI'21 "Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection"

UMS for Multi-turn Response Selection Implements the model described in the following paper Do Response Selection Models Really Know What's Next? Utte

Taesun Whang 47 Nov 22, 2022
Official PyTorch implementation of the paper "Graph-based Generative Face Anonymisation with Pose Preservation" in ICIAP 2021

Contents AnonyGAN Installation Dataset Preparation Generating Images Using Pretrained Model Train and Test New Models Evaluation Acknowledgments Citat

Nicola Dall'Asen 10 May 24, 2022
Codes of the paper Deformable Butterfly: A Highly Structured and Sparse Linear Transform.

Deformable Butterfly: A Highly Structured and Sparse Linear Transform DeBut Advantages DeBut generalizes the square power of two butterfly factor matr

Rui LIN 8 Jun 10, 2022
Neural style transfer in PyTorch.

style-transfer-pytorch An implementation of neural style transfer (A Neural Algorithm of Artistic Style) in PyTorch, supporting CPUs and Nvidia GPUs.

Katherine Crowson 395 Jan 06, 2023
Distinguishing Commercial from Editorial Content in News

Distinguishing Commercial from Editorial Content in News In this repository you can find the following: An anonymized version of the data used for my

Timo Kats 3 Sep 26, 2022
SciKit-Learn Laboratory (SKLL) makes it easy to run machine learning experiments.

SciKit-Learn Laboratory This Python package provides command-line utilities to make it easier to run machine learning experiments with scikit-learn. O

ETS 528 Nov 25, 2022
An implementation of the methods presented in Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.

An implementation of the methods presented in Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.

Andrew Jesson 9 Apr 04, 2022
Time series annotation library.

CrowdCurio Time Series Annotator Library The CrowdCurio Time Series Annotation Library implements classification tasks for time series. Features Suppo

CrowdCurio 51 Sep 15, 2022
AISTATS 2019: Confidence-based Graph Convolutional Networks for Semi-Supervised Learning

Confidence-based Graph Convolutional Networks for Semi-Supervised Learning Source code for AISTATS 2019 paper: Confidence-based Graph Convolutional Ne

MALL Lab (IISc) 56 Dec 03, 2022
DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022; Official code

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism This repository is the official PyTorch implementation of our AAAI-2022 paper, in

Jinglin Liu 803 Dec 28, 2022
Auxiliary Raw Net (ARawNet) is a ASVSpoof detection model taking both raw waveform and handcrafted features as inputs, to balance the trade-off between performance and model complexity.

Overview This repository is an implementation of the Auxiliary Raw Net (ARawNet), which is ASVSpoof detection system taking both raw waveform and hand

6 Jul 08, 2022