Official code for our EMNLP2021 Outstanding Paper MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

Overview

MindCraft

Authors: Cristian-Paul Bara*, Sky CH-Wang*, Joyce Chai

This is the official code repository for the paper (arXiv link):

Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).

@inproceedings{bara2021mindcraft,
  title={MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks},
  author={Bara, Cristian-Paul and CH-Wang, Sky and Chai, Joyce},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2021}
}

Installation Instructions

This README assumes that the user is about to set up the MindCraft task on a to-be-newly-created Ubuntu-based AWS EC2 Server. If not, some commands may be invalid (e.g. apt-get vs. apt). This has been tested on both Ubuntu versions 18 and 20.

Server Setup & Port Forwarding (for reference, AWS port forwarding guidelines are adapted from here):

  1. Launch an EC2 Instance, choosing an Ubuntu-based x86 Amazon Machine Image such as Ubuntu Server 20.04 LTS (HVM), SSD Volume Type.
  2. Choose an instance type. Minimum resource requirements are relatively high, seeing as we are going to run web & game servers concurrently on the same machine. Testing has indicated that instances like t2.xlarge with at least 16 gigs of RAM work fine.
  3. Leave the options specified in 3. Configure Instance, 4. Add Storage, and 5. Add Tags as default.
  4. On 6. Configure Security Group, Add Rule of type Custom TCP Rule, with options Port Range: 22565 and Source: Anywhere. This ensures that players can access the game server with just the IP address.
  5. Add another rule, this time with option Port Range: 8080, keeping all other options the same as above. This ensures that players can access the web server with just the IP address.
  6. Review and launch, specifying your EC2 KeyPair for remote admin access.

Required Depenencies (install these sshed into the EC2 machine):

  1. Java (for reference, Java-Spigot guidelines are adapted from here)
    1. First, update your local package lists with sudo apt update; this updates the URL locations for all the required dependencies you're going to install later. On a newly-created EC2 server, these lists at startup are going to be horribly out of date.
    2. Next, install Java Runtime Environment 8 with sudo apt install openjdk-8-jre-headless.
  2. MySQL (for reference, MySQL guidelines are adapted from here)
    1. To install MySQL server, run sudo apt install mysql-server.
    2. Run setup script sudo mysql_secure_installation to configure your newly installed server with authentication permissions. Go through the setup instructions, setting up a password for the root-user, and confirming other security settings.
    3. To confirm that your newly-installed MySQL server is running, run systemctl status mysql.service to manually check.
    4. Now, the password just created in step 2.2 doesn't actually enable remote connections as root (which we want for our game server & web server); here, execute:
      1. sudo mysql
      2. In the MYSQL commandline, run ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; (replacing password with a password of your choosing).
      3. And finally FLUSH PRIVILEGES; to write changes to disk.
      4. To exit the MySQL commandline, exit.
    5. To create your MySQL database (where all MindCraft user activity is recorded), enter the MySQL commandline this time with mysql -u root -p, and then entering the password you just created.
    6. In the commandline interface, CREATE DATABASE minecraft; to create, and then SHOW DATABASES; to manually confirm the creation of the minecraft database. Before you exit, run SHOW GLOBAL VARIABLES LIKE 'PORT'; to confirm the port that the MySQL server is running on (by default, it should be 3306).
    7. Finally, set up game server plugin authentication details! Before you proceed: if this is your first time setting this up, the files listed below may not have been created yet. To create these files, go ahead and start the server once with bash startServer.sh. You're going to see a ton of errors appear, but hold out for now! When the server is done spinning up, enter stop to stop the server, and then proceed onto these sub-steps.
      1. Open spigot/plugins/situatedDialogue/config.yml, and change:
        1. mysql_password to what you defined in 2.4.2.
        2. mysql_port to what you just confirmed in 2.4.6 (default is 3306).
      2. Open spigot/plugins/LogBlock/config.yml, and change the same lines as above, this time under section mysql. Remember to also change the MySQL user here to root (default initialization has it as something else)!
      3. Open mean/server.js and do the same for MYSQL authentication credentials in the first few lines.
      4. Open spigot/plugins/AdvancedReplay and do the same for mysql.yml.
  3. NPM (for reference, Node.JS guidelines are adapted from here)
    1. Package lists should already be up-to-date, so running sudo apt install nodejs and sudo apt install npm will suffice; this will install the latest versions.
  4. MongoDB (for reference, Mongo guidelines are adapted from here) These steps are necessary mainly because I have used a cookie cutter MEAN stack setup, even though the underlying web server doesn't really use Mongo at all.
    1. Run wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
    2. Run echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
    3. Run sudo apt update and sudo apt install -y mongodb-org to install Mongo.
    4. Run sudo systemctl start mongod to start the service and sudo systemctl status mongod to check service status.
    5. Finally, run sudo systemctl enable mongod to enable Mongo to start on every server reboot.

File Structure

The MindCraft environment is split into three main modules: (1) initialization, by which task variables (e.g. number of games, complexity of games, and more) are set to customized or default values; (2) the game server, a Bukkit/Spigot-made Minecraft multiplayer server that hosts the game itself and records all in-game interactions; and (3) the web server, a MEAN-stack (mainly, Node.JS) web server that records webpage user interactions, where currently recording of player mental states takes place.

All recording of user data -both game server and through the web server- are consolidated into a local MySQL database, the authentication details of which are to be specifiied in initialization files (or left as default).

Execution

Both the game server and web server are designed to be run concurrently in parallel. To achieve this, set up two tmuxes and run the following sections in separate muxes.

Game Server: Run bash startServer.sh. Edit the parameters passed to the plan generator python script if desired before running the server. If it's your first time running the server, you may have to go the newly-generated spigot/eula.txt and change eula=false to eula=true.
Web Server: Navigate to the mean folder and run npm start. Do this after the game server has been successfully spun up.

Replay

For replaying a specific previous game, make sure that the correct plan file of the logged game, indicated in the format logs/logs.XXXXXXX.plan.json (where XXX is the UNIX time stamp of when the game was played), is copied to the following folder and named as plan_generator/plan.json. After this has been done, start the server with bash spigot/start.command instead of the usual bash file.

Owner
Situated Language and Embodied Dialogue (SLED) Research Group
SLED Research Group @ University of Michigan
Situated Language and Embodied Dialogue (SLED) Research Group
PyTorch implementation of Trust Region Policy Optimization

PyTorch implementation of TRPO Try my implementation of PPO (aka newer better variant of TRPO), unless you need to you TRPO for some specific reasons.

Ilya Kostrikov 366 Nov 15, 2022
Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters.

Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters. Overview This project is a Torch implementation for our CVPR 2016 paper

Jianwei Yang 278 Dec 25, 2022
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Jiaxi Jiang 282 Jan 02, 2023
[CVPR'2020] DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

DeepDeform (CVPR'2020) DeepDeform is an RGB-D video dataset containing over 390,000 RGB-D frames in 400 videos, with 5,533 optical and scene flow imag

Aljaz Bozic 165 Jan 09, 2023
BRepNet: A topological message passing system for solid models

BRepNet: A topological message passing system for solid models This repository contains the an implementation of BRepNet: A topological message passin

Autodesk AI Lab 42 Dec 30, 2022
You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors

You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors In this paper, we propose a novel local descriptor-based fra

Haiping Wang 80 Dec 15, 2022
Unofficial PyTorch Implementation of "DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features"

Pytorch Implementation of Deep Orthogonal Fusion of Local and Global Features (DOLG) This is the unofficial PyTorch Implementation of "DOLG: Single-St

DK 96 Jan 06, 2023
Kindle is an easy model build package for PyTorch.

Kindle is an easy model build package for PyTorch. Building a deep learning model became so simple that almost all model can be made by copy and paste from other existing model codes. So why code? wh

Jongkuk Lim 77 Nov 11, 2022
Datasets, Transforms and Models specific to Computer Vision

vision Datasets, Transforms and Models specific to Computer Vision Installation First install the nightly version of OneFlow python3 -m pip install on

OneFlow 68 Dec 07, 2022
Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Make-A-Scene - PyTorch Pytorch implementation (inofficial) of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (https://arxiv.org/

Casual GAN Papers 259 Dec 28, 2022
This is a demo app to be used in the video streaming applications

MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks MoViDNN is an Android application that can be used to ev

ATHENA Christian Doppler (CD) Laboratory 7 Jul 21, 2022
Frequency Spectrum Augmentation Consistency for Domain Adaptive Object Detection

Frequency Spectrum Augmentation Consistency for Domain Adaptive Object Detection Main requirements torch = 1.0 torchvision = 0.2.0 Python 3 Environm

15 Apr 04, 2022
AFL binary instrumentation

E9AFL --- Binary AFL E9AFL inserts American Fuzzy Lop (AFL) instrumentation into x86_64 Linux binaries. This allows binaries to be fuzzed without the

242 Dec 12, 2022
This repo contains the implementation of the algorithm proposed in Off-Belief Learning, ICML 2021.

Off-Belief Learning Introduction This repo contains the implementation of the algorithm proposed in Off-Belief Learning, ICML 2021. Environment Setup

Facebook Research 32 Jan 05, 2023
This app is a simple example of using Strealit to create a financial data web app.

Streamlit Demo: Finance Chart This app is a simple example of using Streamlit to create a financial data web app. This demo use streamlit, pandas and

91 Jan 02, 2023
Official Python implementation of the FuzionCoin protocol

PyFuzc Official Python implementation of the FuzionCoin protocol WARNING: Under construction. Use at your own risk. Some functions may not work. Setup

FuzionCoin 3 Jul 07, 2022
[NeurIPS 2021] Introspective Distillation for Robust Question Answering

Introspective Distillation (IntroD) This repository is the Pytorch implementation of our paper "Introspective Distillation for Robust Question Answeri

Yulei Niu 13 Jul 26, 2022
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
Official Pytorch implementation for video neural representation (NeRV)

NeRV: Neural Representations for Videos (NeurIPS 2021) Project Page | Paper | UVG Data Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav S

hao 214 Dec 28, 2022
Codes for [NeurIPS'21] You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership.

You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership Codes for [NeurIPS'21] You are caught stealing my winni

VITA 8 Nov 01, 2022