Official code for our EMNLP2021 Outstanding Paper MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

Overview

MindCraft

Authors: Cristian-Paul Bara*, Sky CH-Wang*, Joyce Chai

This is the official code repository for the paper (arXiv link):

Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).

@inproceedings{bara2021mindcraft,
  title={MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks},
  author={Bara, Cristian-Paul and CH-Wang, Sky and Chai, Joyce},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2021}
}

Installation Instructions

This README assumes that the user is about to set up the MindCraft task on a to-be-newly-created Ubuntu-based AWS EC2 Server. If not, some commands may be invalid (e.g. apt-get vs. apt). This has been tested on both Ubuntu versions 18 and 20.

Server Setup & Port Forwarding (for reference, AWS port forwarding guidelines are adapted from here):

  1. Launch an EC2 Instance, choosing an Ubuntu-based x86 Amazon Machine Image such as Ubuntu Server 20.04 LTS (HVM), SSD Volume Type.
  2. Choose an instance type. Minimum resource requirements are relatively high, seeing as we are going to run web & game servers concurrently on the same machine. Testing has indicated that instances like t2.xlarge with at least 16 gigs of RAM work fine.
  3. Leave the options specified in 3. Configure Instance, 4. Add Storage, and 5. Add Tags as default.
  4. On 6. Configure Security Group, Add Rule of type Custom TCP Rule, with options Port Range: 22565 and Source: Anywhere. This ensures that players can access the game server with just the IP address.
  5. Add another rule, this time with option Port Range: 8080, keeping all other options the same as above. This ensures that players can access the web server with just the IP address.
  6. Review and launch, specifying your EC2 KeyPair for remote admin access.

Required Depenencies (install these sshed into the EC2 machine):

  1. Java (for reference, Java-Spigot guidelines are adapted from here)
    1. First, update your local package lists with sudo apt update; this updates the URL locations for all the required dependencies you're going to install later. On a newly-created EC2 server, these lists at startup are going to be horribly out of date.
    2. Next, install Java Runtime Environment 8 with sudo apt install openjdk-8-jre-headless.
  2. MySQL (for reference, MySQL guidelines are adapted from here)
    1. To install MySQL server, run sudo apt install mysql-server.
    2. Run setup script sudo mysql_secure_installation to configure your newly installed server with authentication permissions. Go through the setup instructions, setting up a password for the root-user, and confirming other security settings.
    3. To confirm that your newly-installed MySQL server is running, run systemctl status mysql.service to manually check.
    4. Now, the password just created in step 2.2 doesn't actually enable remote connections as root (which we want for our game server & web server); here, execute:
      1. sudo mysql
      2. In the MYSQL commandline, run ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; (replacing password with a password of your choosing).
      3. And finally FLUSH PRIVILEGES; to write changes to disk.
      4. To exit the MySQL commandline, exit.
    5. To create your MySQL database (where all MindCraft user activity is recorded), enter the MySQL commandline this time with mysql -u root -p, and then entering the password you just created.
    6. In the commandline interface, CREATE DATABASE minecraft; to create, and then SHOW DATABASES; to manually confirm the creation of the minecraft database. Before you exit, run SHOW GLOBAL VARIABLES LIKE 'PORT'; to confirm the port that the MySQL server is running on (by default, it should be 3306).
    7. Finally, set up game server plugin authentication details! Before you proceed: if this is your first time setting this up, the files listed below may not have been created yet. To create these files, go ahead and start the server once with bash startServer.sh. You're going to see a ton of errors appear, but hold out for now! When the server is done spinning up, enter stop to stop the server, and then proceed onto these sub-steps.
      1. Open spigot/plugins/situatedDialogue/config.yml, and change:
        1. mysql_password to what you defined in 2.4.2.
        2. mysql_port to what you just confirmed in 2.4.6 (default is 3306).
      2. Open spigot/plugins/LogBlock/config.yml, and change the same lines as above, this time under section mysql. Remember to also change the MySQL user here to root (default initialization has it as something else)!
      3. Open mean/server.js and do the same for MYSQL authentication credentials in the first few lines.
      4. Open spigot/plugins/AdvancedReplay and do the same for mysql.yml.
  3. NPM (for reference, Node.JS guidelines are adapted from here)
    1. Package lists should already be up-to-date, so running sudo apt install nodejs and sudo apt install npm will suffice; this will install the latest versions.
  4. MongoDB (for reference, Mongo guidelines are adapted from here) These steps are necessary mainly because I have used a cookie cutter MEAN stack setup, even though the underlying web server doesn't really use Mongo at all.
    1. Run wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
    2. Run echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
    3. Run sudo apt update and sudo apt install -y mongodb-org to install Mongo.
    4. Run sudo systemctl start mongod to start the service and sudo systemctl status mongod to check service status.
    5. Finally, run sudo systemctl enable mongod to enable Mongo to start on every server reboot.

File Structure

The MindCraft environment is split into three main modules: (1) initialization, by which task variables (e.g. number of games, complexity of games, and more) are set to customized or default values; (2) the game server, a Bukkit/Spigot-made Minecraft multiplayer server that hosts the game itself and records all in-game interactions; and (3) the web server, a MEAN-stack (mainly, Node.JS) web server that records webpage user interactions, where currently recording of player mental states takes place.

All recording of user data -both game server and through the web server- are consolidated into a local MySQL database, the authentication details of which are to be specifiied in initialization files (or left as default).

Execution

Both the game server and web server are designed to be run concurrently in parallel. To achieve this, set up two tmuxes and run the following sections in separate muxes.

Game Server: Run bash startServer.sh. Edit the parameters passed to the plan generator python script if desired before running the server. If it's your first time running the server, you may have to go the newly-generated spigot/eula.txt and change eula=false to eula=true.
Web Server: Navigate to the mean folder and run npm start. Do this after the game server has been successfully spun up.

Replay

For replaying a specific previous game, make sure that the correct plan file of the logged game, indicated in the format logs/logs.XXXXXXX.plan.json (where XXX is the UNIX time stamp of when the game was played), is copied to the following folder and named as plan_generator/plan.json. After this has been done, start the server with bash spigot/start.command instead of the usual bash file.

Owner
Situated Language and Embodied Dialogue (SLED) Research Group
SLED Research Group @ University of Michigan
Situated Language and Embodied Dialogue (SLED) Research Group
MapReader: A computer vision pipeline for the semantic exploration of maps at scale

MapReader A computer vision pipeline for the semantic exploration of maps at scale MapReader is an end-to-end computer vision (CV) pipeline designed b

Living with Machines 25 Dec 26, 2022
PyTorch implementation of Convolutional Neural Fabrics http://arxiv.org/abs/1606.02492

PyTorch implementation of Convolutional Neural Fabrics arxiv:1606.02492 There are some minor differences: The raw image is first convolved, to obtain

Anuvabh Dutt 25 Dec 22, 2021
Implementation of the ICCV'21 paper Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases

Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases [Papers 1, 2][Project page] [Video] The implementation of the papers Temporal

56 Nov 21, 2022
Training RNNs as Fast as CNNs

News SRU++, a new SRU variant, is released. [tech report] [blog] The experimental code and SRU++ implementation are available on the dev branch which

ASAPP Research 2.1k Jan 01, 2023
BEAS: Blockchain Enabled Asynchronous & Secure Federated Machine Learning

BEAS Blockchain Enabled Asynchronous and Secure Federated Machine Learning Default Network Configuration: The default application uses the HyperLedger

Harpreet Virk 11 Nov 20, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 05, 2023
It's final year project of Diploma Engineering. This project is based on Computer Vision.

Face-Recognition-Based-Attendance-System It's final year project of Diploma Engineering. This project is based on Computer Vision. Brief idea about ou

Neel 10 Nov 02, 2022
Language models are open knowledge graphs ( non official implementation )

language-models-are-knowledge-graphs-pytorch Language models are open knowledge graphs ( work in progress ) A non official reimplementation of Languag

theblackcat102 132 Dec 18, 2022
A memory-efficient implementation of DenseNets

efficient_densenet_pytorch A PyTorch =1.0 implementation of DenseNets, optimized to save GPU memory. Recent updates Now works on PyTorch 1.0! It uses

Geoff Pleiss 1.4k Dec 25, 2022
An open-source outlier detection package by Getcontact Data Team

pyfbad The pyfbad library supports anomaly detection projects. An end-to-end anomaly detection application can be written using the source codes of th

Teknasyon Tech 41 Dec 27, 2022
Repo for our ICML21 paper Unsupervised Learning of Visual 3D Keypoints for Control

Unsupervised Learning of Visual 3D Keypoints for Control [Project Website] [Paper] Boyuan Chen1, Pieter Abbeel1, Deepak Pathak2 1UC Berkeley 2Carnegie

Boyuan Chen 34 Jul 22, 2022
Next-Best-View Estimation based on Deep Reinforcement Learning for Active Object Classification

next_best_view_rl Setup Clone the repository: git clone --recurse-submodules ... In 'third_party/zed-ros-wrapper': git checkout devel Install mujoco `

Christian Korbach 1 Feb 15, 2022
My course projects for the 2021 Spring Machine Learning course at the National Taiwan University (NTU)

ML2021Spring There are my projects for the 2021 Spring Machine Learning course at the National Taiwan University (NTU) Course Web : https://speech.ee.

Ding-Li Chen 15 Aug 29, 2022
Adjust Decision Boundary for Class Imbalanced Learning

Adjusting Decision Boundary for Class Imbalanced Learning This repository is the official PyTorch implementation of WVN-RS, introduced in Adjusting De

Peyton Byungju Kim 16 Jan 04, 2023
AWS documentation corpus for zero-shot open-book question answering.

aws-documentation We present the AWS documentation corpus, an open-book QA dataset, which contains 25,175 documents along with 100 matched questions a

Sia Gholami 2 Jul 07, 2022
Python scripts for performing stereo depth estimation using the HITNET Tensorflow model.

HITNET-Stereo-Depth-estimation Python scripts for performing stereo depth estimation using the HITNET Tensorflow model from Google Research. Stereo de

Ibai Gorordo 76 Jan 02, 2023
Generic U-Net Tensorflow implementation for image segmentation

Tensorflow Unet Warning This project is discontinued in favour of a Tensorflow 2 compatible reimplementation of this project found under https://githu

Joel Akeret 1.8k Dec 10, 2022
Code to reproduce the results in the paper "Tensor Component Analysis for Interpreting the Latent Space of GANs".

Tensor Component Analysis for Interpreting the Latent Space of GANs [ paper | project page ] Code to reproduce the results in the paper "Tensor Compon

James Oldfield 4 Jun 17, 2022
Simulate genealogical trees and genomic sequence data using population genetic models

msprime msprime is a population genetics simulator based on tskit. Msprime can simulate random ancestral histories for a sample of individuals (consis

Tskit developers 150 Dec 14, 2022
Paddle-Skeleton-Based-Action-Recognition - DecoupleGCN-DropGraph, ASGCN, AGCN, STGCN

Paddle-Skeleton-Action-Recognition DecoupleGCN-DropGraph, ASGCN, AGCN, STGCN. Yo

Chenxu Peng 3 Nov 02, 2022