Luminous is a framework for testing the performance of Embodied AI (EAI) models in indoor tasks.

Overview

logo

Luminous is a framework for testing the performance of Embodied AI (EAI) models in indoor tasks. Generally, we intergrete different kind of functionalities into this repository that are related to evaluate EAI performance for indoor tasks. The Indoor Scene Synthesis module provides different methods for synthesize randomized indoor scenes that be visualized in Unity Engine. The Luminous for Alfred offers the complete pipline to conduct data augmentment such as generating Alfred tasks in Luminous, solving the trajectories of tasks, rendering images, and generating natural language descripitions.

over

Simulator Layout randomization Small Object Randomization Object Material randomization Number of rooms Number of objects
Habitat x x x 120 1000+?
Virtualhome x x 7 (houses) 357
ThreeDworld x 100+? 2500
iGibson x x 106 (houses) 1984
AI2Thor x 227 2000
Luminous + AI2Thor 2000

Feature Highlights

We highlight the features of Luminous here and readers may go directly into the modules for detailed implementations.

1. Indoor scene synthesis

We present Constrainted Stochastic Scene Generation in Luminous, which generates high-quality scene from the a graph-based representation of indoor scenes. By specifying the required objects and relationships between objects, we can sample an unlimited number of indoor scenes that satisfy the constraints.

teaser

Algorithm Scene Graph Inference Scene Generation Constrained RGBD rendering
PlanIT x x x
Grains x
3DSLN x
Human-Centric x x x
CSSG

2. Luminous for Alfred

We can generate Alfred trajectories from Luminous. After taking the task descriptions in Alfred, we apply the Task Parser to get the required indoor scenes. Then, a Task Sampler sample the tasks for Alfred challenge. Finally, we can add Natural Language Descriptions along with agent actions.

3. EAI Evaluation

We evaluation three models on Luminous for Alfred challenge.

  • MOCA
  • ET
  • HiTUT

See EAIEvaluation folder for more details.

alfred sample_alfred

Folder structure

Luminous
│   readme.md   
│
└─── EAIEvaluation
│   │   ET
│   │   MOCA
│   │   HiTUT
└───IndoorSceneSynthesis
│   │   3DFrontToolBox
│   │   ContrainedStochsticIndoorScene
│   │   LearningBasedMethod
│   │   └───3DSLN
│   │   └───DeepSynth
└───Luminous4Alfred
│   │   NaturalLanguageTool
│   │   TaskParser
│   │   TaskSampler
│   │   TaskSolver

Quick links

EAIEvaluation

CSSG

NaturalLanguageTool

3DFrontToolBox

A Structured Self-attentive Sentence Embedding

Structured Self-attentive sentence embeddings Implementation for the paper A Structured Self-Attentive Sentence Embedding, which was published in ICLR

Kaushal Shetty 488 Nov 28, 2022
Pytorch implementation of CoCon: A Self-Supervised Approach for Controlled Text Generation

COCON_ICLR2021 This is our Pytorch implementation of COCON. CoCon: A Self-Supervised Approach for Controlled Text Generation (ICLR 2021) Alvin Chan, Y

alvinchangw 79 Dec 18, 2022
OCR Streamlit App is used to extract text from images using python's easyocr, pytorch and streamlit packages

OCR-Streamlit-App OCR Streamlit App is used to extract text from images using python's easyocr, pytorch and streamlit packages OCR app gets an image a

Siva Prakash 5 Apr 05, 2022
Code for the KDD 2021 paper 'Filtration Curves for Graph Representation'

Filtration Curves for Graph Representation This repository provides the code from the KDD'21 paper Filtration Curves for Graph Representation. Depende

Machine Learning and Computational Biology Lab 16 Oct 16, 2022
Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Perceiver This Python package implements Perceiver: General Perception with Iterative Attention by Andrew Jaegle in TensorFlow. This model builds on t

Rishit Dagli 84 Oct 15, 2022
​TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.

TextWorld A text-based game generator and extensible sandbox learning environment for training and testing reinforcement learning (RL) agents. Also ch

Microsoft 983 Dec 23, 2022
An energy estimator for eyeriss-like DNN hardware accelerator

Energy-Estimator-for-Eyeriss-like-Architecture- An energy estimator for eyeriss-like DNN hardware accelerator This is an energy estimator for eyeriss-

HEXIN BAO 2 Mar 26, 2022
Extreme Rotation Estimation using Dense Correlation Volumes

Extreme Rotation Estimation using Dense Correlation Volumes This repository contains a PyTorch implementation of the paper: Extreme Rotation Estimatio

Ruojin Cai 29 Nov 18, 2022
LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation

LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation by Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zh

Payphone 8 Nov 21, 2022
Benchmarks for Object Detection in Aerial Images

Benchmarks for Object Detection in Aerial Images

Jian Ding 691 Dec 30, 2022
A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks without the use of any outside machine learning libraries - all from scratch.

Kordel K. France 2 Nov 14, 2022
🔥RandLA-Net in Tensorflow (CVPR 2020, Oral & IEEE TPAMI 2021)

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (CVPR 2020) This is the official implementation of RandLA-Net (CVPR2020, Oral

Qingyong 1k Dec 30, 2022
This is the repository of shape matching algorithm Iterative Rotations and Assignments (IRA)

Description This is the repository of shape matching algorithm Iterative Rotations and Assignments (IRA), described in the publication [1]. Directory

MAMMASMIAS Consortium 6 Nov 14, 2022
A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models with machine learning (NeurIPS 2021 Datasets and Benchmarks Track)

ClimART - A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models Official PyTorch Implementation Using deep le

21 Dec 31, 2022
Do Neural Networks for Segmentation Understand Insideness?

This is part of the code to reproduce the results of the paper Do Neural Networks for Segmentation Understand Insideness? [pdf] by K. Villalobos (*),

biolins 0 Mar 20, 2021
This is the repository of the NeurIPS 2021 paper "Curriculum Disentangled Recommendation withNoisy Multi-feedback"

Curriculum_disentangled_recommendation This is the repository of the NeurIPS 2021 paper "Curriculum Disentangled Recommendation with Noisy Multi-feedb

14 Dec 20, 2022
IGCN : Image-to-graph convolutional network

IGCN : Image-to-graph convolutional network IGCN is a learning framework for 2D/3D deformable model registration and alignment, and shape reconstructi

Megumi Nakao 7 Oct 27, 2022
The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Published by SpaceML • About SpaceML • Quick Colab Example Self-Supervised Learner The Self-Supervised Learner can be used to train a classifier with

SpaceML 92 Nov 30, 2022
Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation(Unfinished)

Keras-FCN Fully convolutional networks and semantic segmentation with Keras. Models Models are found in models.py, and include ResNet and DenseNet bas

645 Dec 29, 2022
Simple codebase for flexible neural net training

neural-modular Simple codebase for flexible neural net training. Allows for seamless exchange of models, dataset, and optimizers. Uses hydra for confi

Jannik Kossen 7 Apr 05, 2022