Create time-series datacubes for supervised machine learning with ICEYE SAR images.

Related tags

Deep Learningicecube
Overview

ICEcube is a Python library intended to help organize SAR images and annotations for supervised machine learning applications. The library generates multidimensional SAR image and labeled data arrays.

The datacubes stack SAR time-series images in range and azimuth and can preserve the geospatial content, intensity, and complex SAR signal from the ICEYE SAR images. You can use the datacubes with ICEYE Ground Range Detected (GRD) geotifs and ICEYE Single Look Complex (SLC) .hdf5 product formats.

alt text

This work is sponsored by ESA Φ-lab as part of the AI4SAR initiative.


Getting Started

You need Python 3.8 or later to use the ICEcube library.

The installation options depend on whether you want to use the library in your Python scripts or you want to contribute to it. For more information, see Installation.


ICEcube Examples

To test the Jupyter notebooks and for information on how to use the library, see the ICEcube Documentation.


AI4SAR Project Updates

For the latest project updates, see SAR for AI Development.

Comments
  • 'RPC' does not exist

    'RPC' does not exist

    Trying to read an SLC .h5 downloaded from ICEYE archive (id 10499) and get 'RPC does not exist':

    cube_config = CubeConfig()
    slc_datacube = SLCDatacube.build(cube_config, '/Users/sstrong/bin/test_data_icecube/slcs')
    
    ---------------------------------------------------------------------------
    KeyError                                  Traceback (most recent call last)
    /var/folders/7r/fyfh8zx51ls6yt8t_jppnz3c0000gq/T/ipykernel_11546/2087236712.py in <module>
          1 cube_config = CubeConfig()
    ----> 2 slc_datacube = SLCDatacube.build(cube_config, '/Users/sstrong/bin/test_data_icecube/slcs')
    
    ~/Documents/github/icecube/icecube/bin/sar_cube/slc_datacube.py in build(cls, cube_config, raster_dir)
         52     def build(cls, cube_config: CubeConfig, raster_dir: str) -> SARDatacube:
         53         slc_datacube = SLCDatacube(cube_config, RASTER_DTYPE)
    ---> 54         ds = slc_datacube.create(cls.PRODUCT_TYPE, raster_dir)
         55         slc_datacube.xrdataset = ds
         56         return slc_datacube
    
    ~/Documents/github/icecube/icecube/utils/common_utils.py in time_it(*args, **kwargs)
        111     def time_it(*args, **kwargs):
        112         time_started = time.time()
    --> 113         return_value = func(*args, **kwargs)
        114         time_elapsed = time.time()
        115         logger.info(
    
    ~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube.py in create(self, product_type, raster_dir)
         43         """
         44         metadata_object = SARDatacubeMetadata(self.cube_config)
    ---> 45         metadata_object = metadata_object.compute_metdatadf_from_folder(
         46             raster_dir, product_type
         47         )
    
    ~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube_metadata.py in compute_metdatadf_from_folder(self, raster_dir, product_type)
        116         )
        117 
    --> 118         self.metadata_df = self._crawl_metadata(raster_dir, product_type)
        119         logger.debug(f"length metadata from the directory {len(self.metadata_df)}")
        120 
    
    ~/Documents/github/icecube/icecube/bin/sar_cube/sar_datacube_metadata.py in _crawl_metadata(self, raster_dir, product_type)
         68 
         69     def _crawl_metadata(self, raster_dir, product_type):
    ---> 70         return metadata_crawler(
         71             raster_dir,
         72             product_type,
    
    ~/Documents/github/icecube/icecube/utils/metadata_crawler.py in metadata_crawler(raster_dir, product_type, variables, recursive)
         36     _, raster_paths = DirUtils.get_dir_files(raster_dir, fext=fext)
         37 
    ---> 38     return metadata_crawler_list(raster_paths, variables)
         39 
         40 
    
    ~/Documents/github/icecube/icecube/utils/metadata_crawler.py in metadata_crawler_list(raster_paths, variables)
         43 
         44     for indx, raster_path in enumerate(raster_paths):
    ---> 45         metadata = IO.load_ICEYE_metadata(raster_path)
         46         parsed_metadata = _parse_data_row(metadata, variables)
         47         parsed_metadata["product_fpath"] = raster_path
    
    ~/Documents/github/icecube/icecube/utils/analytics_IO.py in load_ICEYE_metadata(path)
        432         are converted from bytedata and read into the dict for compatability reasons.
        433         """
    --> 434         return read_SLC_metadata(h5py.File(path, "r"))
        435 
        436     elif path.endswith(".tif") or path.endswith(".tiff"):
    
    ~/Documents/github/icecube/icecube/utils/analytics_IO.py in read_SLC_metadata(h5_io)
        329 
        330     # RPCs are nested under "RPC/" in the h5 thus need to be parsed in a specific manner
    --> 331     RPC_source = h5_io["RPC"]
        332     meta_dict["RPC"] = parse_slc_rpc_to_meta_dict(
        333         RPC_source=RPC_source, meta_dict=meta_dict
    
    h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
    
    h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
    
    /opt/homebrew/anaconda3/envs/icecube_env/lib/python3.8/site-packages/h5py/_hl/group.py in __getitem__(self, name)
        303                 raise ValueError("Invalid HDF5 object reference")
        304         elif isinstance(name, (bytes, str)):
    --> 305             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
        306         else:
        307             raise TypeError("Accessing a group is done with bytes or str, "
    
    h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
    
    h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
    
    h5py/h5o.pyx in h5py.h5o.open()
    
    KeyError: "Unable to open object (object 'RPC' doesn't exist)"
    
    opened by shaystrong 3
  • scikit-image dependency  fails on OSX M1 chip

    scikit-image dependency fails on OSX M1 chip

    Can't install all requirements for icecube on an M1 chip. This may present a future problem, just documenting for awareness. scikit-image cannot seem to be compiled/installed/etc on the M1. I have not tested the conda installation, as perhaps that does work. But i use brew/pip (and conda can create conflicts with those)

    opened by shaystrong 2
  • Fix/labels coords

    Fix/labels coords

    Summary includes:

    • Making xr.dataset structure coherent for labels and SAR (added time coords for labels)
    • For labels datacube, product_fpath are used compared to previously
    • small typo fixed
    • tests added for merging sar cubes with labels cube
    • instructions/cell added to install ml requirements for notebook#5
    • release notes added to mkdocs
    • steup.py updated with ml requirements and version
    opened by muaali 1
  • Update/docs/notebooks

    Update/docs/notebooks

    Changes involve:

    • Introduced a new markdown file called "overview.md" that talks about the structure of examples under docs/
    • Added a new notebook : CreatingDatacube that walks a user how to create datacubes with different methods
    • Other notebooks updated and improved.
    opened by muaali 1
  • missing RPC metadata set to None

    missing RPC metadata set to None

    related to issue: https://github.com/iceye-ltd/icecube/issues/11 Some of old ICEYE images can have RPC information missing. If that happens RPC key will be missing and pipeline does not work. RPC is now set to None if it's missing with a user warning generated.

    opened by muaali 0
  • feat/general metadata

    feat/general metadata

    Following changes introduced:

    • metadata constraints loosen up to allow merging general SAR data (rasterio/HDF5 compatible). But this means that cube configuration is not available for such rasters
    • .tiff support added for GRDs
    • code refactoring in SARDatacubeMetadata to avoid repetitive code
    opened by muaali 0
  • Labels/subset support

    Labels/subset support

    Changes include:

    • Updating SLC metadata reader to avoid key values stored as HDF5 dataset
    • Enabling cube generation from labels.json that have masks/labels for subset rasters (i.e., number of masks ingested into labels cube don't necessarily have to be same as number of rasters)
    • CHUNK_SIZE have been reduced to provide more optimized performance for creating massive datacubes
    opened by muaali 0
  • bin module not found

    bin module not found

    After installing from github using !pip install git+https://github.com/iceye-ltd/icecube.git it imports well icecube, but it throws this error for module bin ModuleNotFoundError: No module named 'icecube.bin'

    Any advice, thanks

    opened by jaimebayes 0
  • dummy_mask_labels.json

    dummy_mask_labels.json

    FileNotFoundError: [Errno 2] No such file or directory: './resources/labels/dummy_mask_labels.json'

    Could you upload it? is it available? Thanks in advance,

    opened by jaimebayes 0
Releases(1.1.0)
Owner
ICEYE Ltd
ICEYE Ltd
ICEYE Ltd
Confident Semantic Ranking Loss for Part Parsing

Confident Semantic Ranking Loss for Part Parsing

Jiachen Xu 5 Oct 22, 2022
Nsdf: A mesh SDF with just some code we can directly paste into our raymarcher

nsdf Representing SDFs of arbitrary meshes has been a bit tricky so far. Express

Jan Ivanecky 5 Feb 18, 2022
PyTorch Implementation of [1611.06440] Pruning Convolutional Neural Networks for Resource Efficient Inference

PyTorch implementation of [1611.06440 Pruning Convolutional Neural Networks for Resource Efficient Inference] This demonstrates pruning a VGG16 based

Jacob Gildenblat 836 Dec 26, 2022
Autonomous Ground Vehicle Navigation and Control Simulation Examples in Python

Autonomous Ground Vehicle Navigation and Control Simulation Examples in Python THIS PROJECT IS CURRENTLY A WORK IN PROGRESS AND THUS THIS REPOSITORY I

Joshua Marshall 14 Dec 31, 2022
DIVeR: Deterministic Integration for Volume Rendering

DIVeR: Deterministic Integration for Volume Rendering This repo contains the training and evaluation code for DIVeR. Setup python 3.8 pytorch 1.9.0 py

64 Dec 27, 2022
Library to enable Bayesian active learning in your research or labeling work.

Bayesian Active Learning (BaaL) BaaL is an active learning library developed at ElementAI. This repository contains techniques and reusable components

ElementAI 687 Dec 25, 2022
Rename Images with Auto Generated Neural Image Captions

Recaption Images with Generated Neural Image Caption Example Usage: Commandline: Recaption all images from folder /home/feng/Downloads/images to folde

feng wang 3 May 01, 2022
DrQ-v2: Improved Data-Augmented Reinforcement Learning

DrQ-v2: Improved Data-Augmented RL Agent Method DrQ-v2 is a model-free off-policy algorithm for image-based continuous control. DrQ-v2 builds on DrQ,

Facebook Research 234 Jan 01, 2023
Fast RFC3339 compliant Python date-time library

udatetime: Fast RFC3339 compliant date-time library Handling date-times is a painful act because of the sheer endless amount of formats used by people

Simon Pirschel 235 Oct 25, 2022
This repository contains the database and code used in the paper Embedding Arithmetic for Text-driven Image Transformation

This repository contains the database and code used in the paper Embedding Arithmetic for Text-driven Image Transformation (Guillaume Couairon, Holger

Meta Research 31 Oct 17, 2022
Pytorch implementation of Nueral Style transfer

Nueral Style Transfer Pytorch implementation of Nueral style transfer algorithm , it is used to apply artistic styles to content images . Content is t

Abhinav 9 Oct 15, 2022
Solving Zero-Shot Learning in Named Entity Recognition with Common Sense Knowledge

Zero-Shot Learning in Named Entity Recognition with Common Sense Knowledge Associated code for the paper Zero-Shot Learning in Named Entity Recognitio

Søren Hougaard Mulvad 13 Dec 25, 2022
A Machine Teaching Framework for Scalable Recognition

MEMORABLE This repository contains the source code accompanying our ICCV 2021 paper. A Machine Teaching Framework for Scalable Recognition Pei Wang, N

2 Dec 08, 2021
Code for the Shortformer model, from the paper by Ofir Press, Noah A. Smith and Mike Lewis.

Shortformer This repository contains the code and the final checkpoint of the Shortformer model. This file explains how to run our experiments on the

Ofir Press 138 Apr 15, 2022
The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

Qi Fan 46 Nov 17, 2022
Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces"

Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces" This repo contains the implementation of GEBO algorithm.

Jaeyeon Ahn 2 Mar 22, 2022
HyperDict - Self linked dictionary in Python

Hyper Dictionary Advanced python dictionary(hash-table), which can link it-self

8 Feb 06, 2022
This is the repository of our article published on MDPI Entropy "Feature Selection for Recommender Systems with Quantum Computing".

Collaborative-driven Quantum Feature Selection This repository was developed by Riccardo Nembrini, PhD student at Politecnico di Milano. See the websi

Quantum Computing Lab @ Politecnico di Milano 10 Apr 21, 2022
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Pranav 39 Nov 21, 2022
Source code of the paper Meta-learning with an Adaptive Task Scheduler.

ATS About Source code of the paper Meta-learning with an Adaptive Task Scheduler. If you find this repository useful in your research, please cite the

Huaxiu Yao 16 Dec 26, 2022