Python parser for DTED data.

Related tags

Deep Learningdted
Overview

DTED Parser

This is a package written in pure python (with help from numpy) to parse and investigate Digital Terrain Elevation Data (DTED) files. This package is tested to work on Shuttle Radar Topography Mission (SRTM) DTED files (as far as I can tell these are the only publicly available DTED files). This can be used as a library to parse these files into numpy arrays and additionally exposes a CLI that can be used to investigate individual DTED files.

For more information and resources about the DTED file format see the end of the README.

How to install

You can install this as a normal python package using pip

pip install dted

How to use

The following example code will parse DTED file checked into this repository for testing.

As a library

Parsing a DTED file into a numpy array is as simple as:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
assert isinstance(tile.data, np.ndarray)

Additionally you can access the metadata of the DTED file (the User Header Label, Data Set Identification, and Accuracy Description records) easily.

from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
print(tile.dsi.south_west_corner)

Parsing entire DTED files has been heavily optimized, but does still take a little bit of time. On my machine (2014 MacbookPro) parsing the 25MB example file take about 120 ms. However, if you only need to look up specific terrain elevations within a DTED file you do not need to parse the entire file. Doing the following takes <1ms on my machine:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
print(tile.get_elevation(LatLon(latitude=41.5, longitude=-70.5)))

If for some reason you really need to eek out every bit of performance and you thoroughly trust your DTED data, you speed up the data parsing by skipping the checksum verification. Doing the following takes about 75 ms on my machine:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
tile.load_data(perform_checksum=False)

assert isinstance(tile.data, np.ndarray)

The final functionality the dted.Tile class offers is to easily check if a coordinate location is contained within the DTED file. This also does not require that the DTED data is fully loaded into memory:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)

assert LatLon(latitude=41.5, longitude=-70.25) in tile

As a CLI

Installing this package into an activated virtual environment also exposes the dted terminal command. This provides three pieces of functionality:

  1. See report of the metadata of the DTED file.
  2. Lookup terrain elevation at a specific point within the DTED file.
  3. Display and ASCII representation of the DTED file in your terminal.

To get a report of the file metadata:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 
File Path:          test/data/n41_w071_1arc_v3.dt2 (24 MB)
Product Level:      DTED2
Security Code:      U
Compilation Date:   02/2000
Maintenance Date:   
Datums (V/H):       E96/WGS84

    (42.0N,71.0W)      (42.0N,70.0W)
          NW --------------- NE     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          SW --------------- SE     
    (41.0N,71.0W)      (41.0N,70.0W)

Origin:                 (41.0N,71.0W)
Resolution (lat/lon):   1.0"/1.0"
Accuracy (V/H):         6m/13m

To lookup terrain elevation at a specific point:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --location 41.7 -70.4
51.0 meters

To display the DTED file in your terminal:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --display

This will attempt to create an ASCII representation of the DTED file within your terminal at the best resolution possible. Increasing the size of your terminal window or zooming out your terminal window will increase the resolution of the chart:

Normal Resolution Image

High Resolution Image

Why did I add this feature? Why not?

If you want to plot this data like a sane person, you can use the following example code with the matplotlib package (not included)

import matplotlib.pyplot as plt
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
plt.imshow(tile.data.T[::-1], cmap="hot")

The DTED file format

This parser was created using the specification provided here:

https://www.dlr.de/eoc/Portaldata/60/Resources/dokumente/7_sat_miss/SRTM-XSAR-DEM-DTED-1.1.pdf

Some things to be aware of with the DTED file format:

  1. Some DTED files contain "void" values for data points where elevation data is not known (such as over bodies of water). An example of such a file can be found at test/data/n00_e006_3arc_v2.dt1. This package will emit a warning if void data is found, and the definition of the void value can be found in dted.definitions.VOID_DATA_VALUE.
  2. The DTED data is structured along longitudinal lines. Therefore, when accessing the data within the numpy array the rows correspond to longitude and the columns correspond to latitude. This may seem backwards to your intuition, i.e. you would access the elevation at a coordinate point with tile.data[longitude_index, latitude_index].
  3. Elevation within the DTED file is encoded using "signed magnitude" notation. This has no effect on a user of this package interacting with the parsed terrain elevation data, but it does slow down the parsing of this data as I do not know of an optimized method of parsing signed magnitude data in python. If someone knows how to do this this parsing library could become even faster.

Where to find DTED data

Publicly available DTED data is relatively hard to find and access, but it can be done. The DTED files I used for testing and developing this package come from https://earthexplorer.usgs.gov/.

This EarthExplorer app provided by the USGS provides an interface to download many types of terrain data, including the SRTM DTED data. However, you need to make an account with them in order to perform and I'm unsure of a way to use their machine-to-machine API to automate downloading data.

Contributing

Contributions are absolutely encouraged! To develop on this project you need to install the poetry package manager.

Clone the repo:

[email protected]$ git clone https://github.com/bbonenfant/dted

Create and activate the virtual environment:

[email protected]$ poetry install && source .venv/bin/activate

To run the tests:

(.venv) [email protected]$ pytest .

If you are getting BLACK errors from pytest, run the black code formatter:

(.venv) [email protected]$ black .
Comments
  • Areas above 50° or below -50° Latitude

    Areas above 50° or below -50° Latitude

    Hello,

    I am trying to use the Tile(dted_file) on SRTM 1arc DTED files. The publicly available SRTM 1arc data is actually not 1arc by 1arc but 2arc by 1arc, as soon as you are working with areas above 50° Latitude.

    This causes the Tile() method to fail with the error: dted.errors.InvalidFileError: Checksum failed for data block

    When I try to ignore the checksum with load_data(perform_checksum=False), it fails with the error: dted.errors.InvalidFileError: All data blocks within a DTED file must begin with 170. Found: 1

    Do I just have to do something differently or is this something you could fix? It works great otherwise!

    Thank you for the help.

    opened by StefanBregenzer 10
  • Error in calculating latitude and longitude indices

    Error in calculating latitude and longitude indices

    I receive drastically different elevation values for the same location when using DTED levels 0, 1, and 2. An area that I know to be approximately 735m MSL reports the following:

    ~/Desktop/dted/w117$ dted n34.dt0 --location 34.353932 -116.295523
    1035.0 meters
    ~/Desktop/dted/w117$ dted n34.dt1 --location 34.353932 -116.295523
    2557.0 meters
    ~/Desktop/dted/w117$ dted n34.dt2 --location 34.353932 -116.295523
    733.0 meters
    

    In looking at the source code, I believe the calculations for the latitude and longitude indices in the DTED data are wrong:

    I believe this:

            lat_interval, lon_interval = self.dsi.latitude_interval, self.dsi.longitude_interval
            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) / lat_interval
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1) / lon_interval
            )
    

    Should be this:

            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) 
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1)
            )
    
    opened by rickpresley 2
  • Adding in the Tiles class.

    Adding in the Tiles class.

    New Tiles class added.

    • allows user to pass in directory of dted files, and query that list for elevations
    • functionality works regardless of filenames
    • functionality works recursively in directory
    • works even with a mix of dted 1 and dted 2 files
    • added a description of how to use in the README
    opened by westonCoder 0
  • Import Error

    Import Error

    I am getting an error when importing Tile from dted. I am using a fresh Conda environment but cannot seem to get around this error. I can see the Tile class in dted/dted/tile.py but the import does not work. Error shown below:

    ImportError: cannot import name 'Tile' from partially initialized module 'dted' (most likely due to a circular import) (/<PATH TO FILE>/dted.py)
    
    Python Version: Python 3.9.12
    

    Thanks for your help in advance!

    opened by OliverHeilmann 4
  • Query a DTED Directory instead of a specific File

    Query a DTED Directory instead of a specific File

    It'd be very useful to have a directory full of dted files, and be able to query the pool of tiles instead of checking each individual one. It could get fancy with loading tiles as needed, tracking frequency of use to free up older ones, etc.

    enhancement 
    opened by KPB3rd 2
Releases(v1.0.3)
Owner
Ben Bonenfant
Ben Bonenfant
Emotion classification of online comments based on RNN

emotion_classification Emotion classification of online comments based on RNN, the accuracy of the model in the test set reaches 99% data: Large Movie

1 Nov 23, 2021
Gradient representations in ReLU networks as similarity functions

Gradient representations in ReLU networks as similarity functions by Dániel Rácz and Bálint Daróczy. This repo contains the python code related to our

1 Oct 08, 2021
A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Stylegan2-Ada-Google-Colab-Starter-Notebook A no thrills colab notebook for training Stylegan2-ada on colab. transfer learning onto your own dataset h

Harnick Khera 66 Dec 16, 2022
Single-Stage 6D Object Pose Estimation, CVPR 2020

Overview This repository contains the code for the paper Single-Stage 6D Object Pose Estimation. Yinlin Hu, Pascal Fua, Wei Wang and Mathieu Salzmann.

CVLAB @ EPFL 89 Dec 26, 2022
Face Library is an open source package for accurate and real-time face detection and recognition

Face Library Face Library is an open source package for accurate and real-time face detection and recognition. The package is built over OpenCV and us

52 Nov 09, 2022
Towards Multi-Camera 3D Human Pose Estimation in Wild Environment

PanopticStudio Toolbox This repository has a toolbox to download, process, and visualize the Panoptic Studio (Panoptic) data. Note: Sep-21-2020: Curre

335 Jan 09, 2023
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
This library provides an abstraction to perform Model Versioning using Weight & Biases.

Description This library provides an abstraction to perform Model Versioning using Weight & Biases. Features Version a new trained model Promote a mod

Hector Lopez Almazan 2 Jan 28, 2022
Official public repository of paper "Intention Adaptive Graph Neural Network for Category-Aware Session-Based Recommendation"

Intention Adaptive Graph Neural Network (IAGNN) This is the official repository of paper Intention Adaptive Graph Neural Network for Category-Aware Se

9 Nov 22, 2022
Multi-View Radar Semantic Segmentation

Multi-View Radar Semantic Segmentation Paper Multi-View Radar Semantic Segmentation, ICCV 2021. Arthur Ouaknine, Alasdair Newson, Patrick Pérez, Flore

valeo.ai 37 Oct 25, 2022
High-quality implementations of standard and SOTA methods on a variety of tasks.

Uncertainty Baselines The goal of Uncertainty Baselines is to provide a template for researchers to build on. The baselines can be a starting point fo

Google 1.1k Dec 30, 2022
CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms

CARLA - Counterfactual And Recourse Library CARLA is a python library to benchmark counterfactual explanation and recourse models. It comes out-of-the

Carla Recourse 200 Dec 28, 2022
Official implementation of paper "Query2Label: A Simple Transformer Way to Multi-Label Classification".

Introdunction This is the official implementation of the paper "Query2Label: A Simple Transformer Way to Multi-Label Classification". Abstract This pa

Shilong Liu 274 Dec 28, 2022
High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

TL;DR Ignite is a high-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently. Click on the image to

4.2k Jan 01, 2023
Dataset Condensation with Contrastive Signals

Dataset Condensation with Contrastive Signals This repository is the official implementation of Dataset Condensation with Contrastive Signals (DCC). T

3 May 19, 2022
Visualizing lattice vibration information from phonon dispersion to atoms (For GPUMD)

Phonon-Vibration-Viewer (For GPUMD) Visualizing lattice vibration information from phonon dispersion for primitive atoms. In this tutorial, we will in

Liangting 6 Dec 10, 2022
Probabilistic Programming and Statistical Inference in PyTorch

PtStat Probabilistic Programming and Statistical Inference in PyTorch. Introduction This project is being developed during my time at Cogent Labs. The

Stefano Peluchetti 109 Nov 26, 2022
An implementation of based on pytorch and mmcv

FisherPruning-Pytorch An implementation of Group Fisher Pruning for Practical Network Compression based on pytorch and mmcv Main Functions Pruning f

Peng Lu 15 Dec 17, 2022
Apply Graph Self-Supervised Learning methods to graph-level task(TUDataset, MolculeNet Datset)

Graphlevel-SSL Overview Apply Graph Self-Supervised Learning methods to graph-level task(TUDataset, MolculeNet Dataset). It is unified framework to co

JunSeok 8 Oct 15, 2021
PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML)

pytorch-maml This is a PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML): https://arxiv

Kate Rakelly 516 Jan 05, 2023