Neural Nano-Optics for High-quality Thin Lens Imaging

Last update: Dec 05, 2022

Related tags

Overview

Neural Nano-Optics for High-quality Thin Lens Imaging

Project Page | Paper | Data

Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar, Felix Heide

This code implements a differentiable proxy model for simulating meta-optics and a neural feature propagation deconvolution method. These components are optimized end-to-end using machine learning optimizers.

The experimental results from the manuscript and the supplemental information are reproducible with this implementation. The proposed differentiable proxy model, neural feature propagation, and end-to-end optimization framework are implemented completely in TensorFlow, without dependency on third-party libraries.

Training

To perform end-to-end training (of meta-optic and deconvolution) execute the 'run_train.sh' script. The model checkpoint which includes saved parameters for both the meta-optic and deconvolution will be saved to 'training/ckpt'. The folder 'training/data' contains a subset of the training and test data that we used for optimizing our end-to-end imaging pipeline.

Testing

To perform inference on real-world captures launch the "test.ipynb" notebook in Jupyter Notebook and step through the cells. The notebook will load in a finetuned checkpoint of our neural feature propagation network from 'experimental/ckpt' which will process captured sensor measurements located in 'experimental/data'. The reconstructed images will be displayed within the notebook.

Additional captured sensor measurements can be found in the data repository.

Requirements

This code has been tested with Python 3.6.10 using TensorFlow 2.2.0 running on Linux with an Nvidia P100 GPU with 16GB RAM.

We installed the following library packages to run this code:

TensorFlow >= 2.2
TensorFlow Probability
TensorFlow Addons
Numpy
Scipy
matplotlib
jupyter-notebook

Citation

If you find our work useful in your research, please cite:

@article{Tseng2021NeuralNanoOptics,
    title   = "Neural Nano-Optics for High-quality Thin Lens Imaging",
    author  = "Tseng, Ethan and Colburn, Shane and Whitehead, James and Huang, Luocheng
               and Baek, Seung-Hwan and Majumdar, Arka and Heide, Felix",
    journal = "Nature Communications",
    volume  = ,
    number  = ,
    pages   = ,
    year    = 2021
}

License

Our code is licensed under BSL-1. By downloading the software, you agree to the terms of this License. The training data in the folder 'training/data' comes from the INRIA Holidays Dataset.

Comments

I want to ask you some questions.

嗨~ I am studying your article, but for a beginner, there are too many things I don't understand. I want to ask you the details of the specific mapping between the phase function and the scatterer structure.Thank you very much.

opened by Rishell 0
Problem with training

Hello, thank you very much for the code, but I found some problems while using it. At that time when I was training with the source code and data, I found that the result of the generator was all white. Can you tell me what could be the reason for this?

opened by FZfangzheng 0

A Python training and inference implementation of Yolov5 helmet detection in Jetson Xavier nx and Jetson nano

yolov5-helmet-detection-python A Python implementation of Yolov5 to detect head or helmet in the wild in Jetson Xavier nx and Jetson nano. In Jetson X

12 Dec 5, 2022

Predict bus arrival time using VertexAI and Nvidia's Jetson Nano

bus_prediction predict bus arrival time using VertexAI and Nvidia's Jetson Nano imagenet the command for imagenet.py look like this python3 /path/to/i

10 Dec 22, 2022

Implementation of Common Image Evaluation Metrics by Sayed Nadim (sayednadim.github.io). The repo is built based on full reference image quality metrics such as L1, L2, PSNR, SSIM, LPIPS. and feature-level quality metrics such as FID, IS. It can be used for evaluating image denoising, colorization, inpainting, deraining, dehazing etc. where we have access to ground truth.

Image Quality Evaluation Metrics Implementation of some common full reference image quality metrics. The repo is built based on full reference image q

10 Jan 1, 2023

PyZebrascope - an open-source Python platform for brain-wide neural activity imaging in behaving zebrafish

1 May 31, 2022

This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Underwater Light Field Retention : Neural Rendering for Underwater Imaging (UWNR) (Accepted by CVPR Workshop2022 NTIRE) Authors: Tian Ye†, Sixiang Che

17 Dec 14, 2022

This is an official implementation of "Polarized Self-Attention: Towards High-quality Pixel-wise Regression"

Polarized Self-Attention: Towards High-quality Pixel-wise Regression This is an official implementation of: Huajun Liu, Fuqiang Liu, Xinyi Fan and Don

212 Jan 8, 2023

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment This is a pytorch project for the paper Seeing Dynamic Scene i

21 Nov 28, 2022

NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions Overview NUANCED is a user-centric conversational recommen

18 Dec 28, 2021

A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.

About This repository provides data and code for the paper: Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development (subm

86 Dec 7, 2022

Neural Nano-Optics for High-quality Thin Lens Imaging

Related tags

Overview

Neural Nano-Optics for High-quality Thin Lens Imaging

Project Page | Paper | Data

Training

Testing

Requirements

Citation

License

You might also like...

A Python training and inference implementation of Yolov5 helmet detection in Jetson Xavier nx and Jetson nano

Predict bus arrival time using VertexAI and Nvidia's Jetson Nano

PyZebrascope - an open-source Python platform for brain-wide neural activity imaging in behaving zebrafish

This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

This is an official implementation of "Polarized Self-Attention: Towards High-quality Pixel-wise Regression"

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.

Comments

I want to ask you some questions.

Problem with training

Releases(v1.0.0)

v1.0.0(Nov 2, 2021)

Owner

Ethan Tseng

Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM)

An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available actions

The final project of "Applying AI to EHR Data" of "AI for Healthcare" nanodegree - Udacity.

[CVPR 2022 Oral] Balanced MSE for Imbalanced Visual Regression https://arxiv.org/abs/2203.16427

Generate Contextual Directory Wordlist For Target Org

PyTorch implementation of MulMON

TART - A PyTorch implementation for Transition Matrix Representation of Trees with Transposed Convolutions

Code base for "On-the-Fly Test-time Adaptation for Medical Image Segmentation"

Implementation for On Provable Benefits of Depth in Training Graph Convolutional Networks

The implementation of the paper "A Deep Feature Aggregation Network for Accurate Indoor Camera Localization".

Adapter-BERT: Parameter-Efficient Transfer Learning for NLP.

Reimplementation of Learning Mesh-based Simulation With Graph Networks

codes for paper Combining Dynamic Local Context Focus and Dependency Cluster Attention for Aspect-level sentiment classification

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels

Official Implementation of Neural Splines

Code for "Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans" CVPR 2021 best paper candidate

This code provides a PyTorch implementation for OTTER (Optimal Transport distillation for Efficient zero-shot Recognition), as described in the paper.

PyTorch implementation of Neural Dual Contouring.

No-reference Image Quality Assessment(NIQA) Algorithms (BRISQUE, NIQE, PIQE, RankIQA, MetaIQA)