Python parser for DTED data.

Related tags

Deep Learningdted
Overview

DTED Parser

This is a package written in pure python (with help from numpy) to parse and investigate Digital Terrain Elevation Data (DTED) files. This package is tested to work on Shuttle Radar Topography Mission (SRTM) DTED files (as far as I can tell these are the only publicly available DTED files). This can be used as a library to parse these files into numpy arrays and additionally exposes a CLI that can be used to investigate individual DTED files.

For more information and resources about the DTED file format see the end of the README.

How to install

You can install this as a normal python package using pip

pip install dted

How to use

The following example code will parse DTED file checked into this repository for testing.

As a library

Parsing a DTED file into a numpy array is as simple as:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
assert isinstance(tile.data, np.ndarray)

Additionally you can access the metadata of the DTED file (the User Header Label, Data Set Identification, and Accuracy Description records) easily.

from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
print(tile.dsi.south_west_corner)

Parsing entire DTED files has been heavily optimized, but does still take a little bit of time. On my machine (2014 MacbookPro) parsing the 25MB example file take about 120 ms. However, if you only need to look up specific terrain elevations within a DTED file you do not need to parse the entire file. Doing the following takes <1ms on my machine:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
print(tile.get_elevation(LatLon(latitude=41.5, longitude=-70.5)))

If for some reason you really need to eek out every bit of performance and you thoroughly trust your DTED data, you speed up the data parsing by skipping the checksum verification. Doing the following takes about 75 ms on my machine:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
tile.load_data(perform_checksum=False)

assert isinstance(tile.data, np.ndarray)

The final functionality the dted.Tile class offers is to easily check if a coordinate location is contained within the DTED file. This also does not require that the DTED data is fully loaded into memory:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)

assert LatLon(latitude=41.5, longitude=-70.25) in tile

As a CLI

Installing this package into an activated virtual environment also exposes the dted terminal command. This provides three pieces of functionality:

  1. See report of the metadata of the DTED file.
  2. Lookup terrain elevation at a specific point within the DTED file.
  3. Display and ASCII representation of the DTED file in your terminal.

To get a report of the file metadata:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 
File Path:          test/data/n41_w071_1arc_v3.dt2 (24 MB)
Product Level:      DTED2
Security Code:      U
Compilation Date:   02/2000
Maintenance Date:   
Datums (V/H):       E96/WGS84

    (42.0N,71.0W)      (42.0N,70.0W)
          NW --------------- NE     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          SW --------------- SE     
    (41.0N,71.0W)      (41.0N,70.0W)

Origin:                 (41.0N,71.0W)
Resolution (lat/lon):   1.0"/1.0"
Accuracy (V/H):         6m/13m

To lookup terrain elevation at a specific point:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --location 41.7 -70.4
51.0 meters

To display the DTED file in your terminal:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --display

This will attempt to create an ASCII representation of the DTED file within your terminal at the best resolution possible. Increasing the size of your terminal window or zooming out your terminal window will increase the resolution of the chart:

Normal Resolution Image

High Resolution Image

Why did I add this feature? Why not?

If you want to plot this data like a sane person, you can use the following example code with the matplotlib package (not included)

import matplotlib.pyplot as plt
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
plt.imshow(tile.data.T[::-1], cmap="hot")

The DTED file format

This parser was created using the specification provided here:

https://www.dlr.de/eoc/Portaldata/60/Resources/dokumente/7_sat_miss/SRTM-XSAR-DEM-DTED-1.1.pdf

Some things to be aware of with the DTED file format:

  1. Some DTED files contain "void" values for data points where elevation data is not known (such as over bodies of water). An example of such a file can be found at test/data/n00_e006_3arc_v2.dt1. This package will emit a warning if void data is found, and the definition of the void value can be found in dted.definitions.VOID_DATA_VALUE.
  2. The DTED data is structured along longitudinal lines. Therefore, when accessing the data within the numpy array the rows correspond to longitude and the columns correspond to latitude. This may seem backwards to your intuition, i.e. you would access the elevation at a coordinate point with tile.data[longitude_index, latitude_index].
  3. Elevation within the DTED file is encoded using "signed magnitude" notation. This has no effect on a user of this package interacting with the parsed terrain elevation data, but it does slow down the parsing of this data as I do not know of an optimized method of parsing signed magnitude data in python. If someone knows how to do this this parsing library could become even faster.

Where to find DTED data

Publicly available DTED data is relatively hard to find and access, but it can be done. The DTED files I used for testing and developing this package come from https://earthexplorer.usgs.gov/.

This EarthExplorer app provided by the USGS provides an interface to download many types of terrain data, including the SRTM DTED data. However, you need to make an account with them in order to perform and I'm unsure of a way to use their machine-to-machine API to automate downloading data.

Contributing

Contributions are absolutely encouraged! To develop on this project you need to install the poetry package manager.

Clone the repo:

[email protected]$ git clone https://github.com/bbonenfant/dted

Create and activate the virtual environment:

[email protected]$ poetry install && source .venv/bin/activate

To run the tests:

(.venv) [email protected]$ pytest .

If you are getting BLACK errors from pytest, run the black code formatter:

(.venv) [email protected]$ black .
Comments
  • Areas above 50° or below -50° Latitude

    Areas above 50° or below -50° Latitude

    Hello,

    I am trying to use the Tile(dted_file) on SRTM 1arc DTED files. The publicly available SRTM 1arc data is actually not 1arc by 1arc but 2arc by 1arc, as soon as you are working with areas above 50° Latitude.

    This causes the Tile() method to fail with the error: dted.errors.InvalidFileError: Checksum failed for data block

    When I try to ignore the checksum with load_data(perform_checksum=False), it fails with the error: dted.errors.InvalidFileError: All data blocks within a DTED file must begin with 170. Found: 1

    Do I just have to do something differently or is this something you could fix? It works great otherwise!

    Thank you for the help.

    opened by StefanBregenzer 10
  • Error in calculating latitude and longitude indices

    Error in calculating latitude and longitude indices

    I receive drastically different elevation values for the same location when using DTED levels 0, 1, and 2. An area that I know to be approximately 735m MSL reports the following:

    ~/Desktop/dted/w117$ dted n34.dt0 --location 34.353932 -116.295523
    1035.0 meters
    ~/Desktop/dted/w117$ dted n34.dt1 --location 34.353932 -116.295523
    2557.0 meters
    ~/Desktop/dted/w117$ dted n34.dt2 --location 34.353932 -116.295523
    733.0 meters
    

    In looking at the source code, I believe the calculations for the latitude and longitude indices in the DTED data are wrong:

    I believe this:

            lat_interval, lon_interval = self.dsi.latitude_interval, self.dsi.longitude_interval
            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) / lat_interval
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1) / lon_interval
            )
    

    Should be this:

            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) 
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1)
            )
    
    opened by rickpresley 2
  • Adding in the Tiles class.

    Adding in the Tiles class.

    New Tiles class added.

    • allows user to pass in directory of dted files, and query that list for elevations
    • functionality works regardless of filenames
    • functionality works recursively in directory
    • works even with a mix of dted 1 and dted 2 files
    • added a description of how to use in the README
    opened by westonCoder 0
  • Import Error

    Import Error

    I am getting an error when importing Tile from dted. I am using a fresh Conda environment but cannot seem to get around this error. I can see the Tile class in dted/dted/tile.py but the import does not work. Error shown below:

    ImportError: cannot import name 'Tile' from partially initialized module 'dted' (most likely due to a circular import) (/<PATH TO FILE>/dted.py)
    
    Python Version: Python 3.9.12
    

    Thanks for your help in advance!

    opened by OliverHeilmann 4
  • Query a DTED Directory instead of a specific File

    Query a DTED Directory instead of a specific File

    It'd be very useful to have a directory full of dted files, and be able to query the pool of tiles instead of checking each individual one. It could get fancy with loading tiles as needed, tracking frequency of use to free up older ones, etc.

    enhancement 
    opened by KPB3rd 2
Releases(v1.0.3)
Owner
Ben Bonenfant
Ben Bonenfant
The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.

News December 27: v1.1.0 New loss functions: CentroidTripletLoss and VICRegLoss Mean reciprocal rank + per-class accuracies See the release notes Than

Kevin Musgrave 5k Jan 05, 2023
[CVPR2021] Invertible Image Signal Processing

Invertible Image Signal Processing This repository includes official codes for "Invertible Image Signal Processing (CVPR2021)". Figure: Our framework

Yazhou XING 281 Dec 31, 2022
[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF Project Page | Paper | Dataset Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu*, Shangzhan zhang*,

Xiao Fu 111 Dec 16, 2022
Gym Threat Defense

Gym Threat Defense The Threat Defense environment is an OpenAI Gym implementation of the environment defined as the toy example in Optimal Defense Pol

Hampus Ramström 5 Dec 08, 2022
Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are implemented and can be seen in tensorboard.

Sarus published models Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are

Sarus Technologies 39 Aug 19, 2022
Final project for machine learning (CSC 590). Detection of hepatitis C and progression through blood samples.

Hepatitis C Blood Based Detection Final project for machine learning (CSC 590). Dataset from Kaggle. Using data from previous hepatitis C blood panels

Jennefer Maldonado 1 Dec 28, 2021
ICNet for Real-Time Semantic Segmentation on High-Resolution Images, ECCV2018

ICNet for Real-Time Semantic Segmentation on High-Resolution Images by Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, Jiaya Jia, details a

Hengshuang Zhao 594 Dec 31, 2022
Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

WECHSEL Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. arXiv: https://arx

Institute of Computational Perception 45 Dec 29, 2022
YOLOv5 + ROS2 object detection package

YOLOv5-ROS YOLOv5 + ROS2 object detection package This program changes the input of detect.py (ultralytics/yolov5) to sensor_msgs/Image of ROS2. Requi

Ar-Ray 23 Dec 19, 2022
Implementation for Stankevičiūtė et al. "Conformal time-series forecasting", NeurIPS 2021.

Conformal time-series forecasting Implementation for Stankevičiūtė et al. "Conformal time-series forecasting", NeurIPS 2021. If you use our code in yo

Kamilė Stankevičiūtė 36 Nov 21, 2022
Summary of related papers on visual attention

This repo is built for paper: Attention Mechanisms in Computer Vision: A Survey paper Vision-Attention-Papers Channel attention Spatial attention Temp

MenghaoGuo 2.1k Dec 30, 2022
Public implementation of the Convolutional Motif Kernel Network (CMKN) architecture

CMKN Implementation of the convolutional motif kernel network (CMKN) introduced in Ditz et al., "Convolutional Motif Kernel Network", 2021. Testing Yo

1 Nov 17, 2021
Learning and Building Convolutional Neural Networks using PyTorch

Image Classification Using Deep Learning Learning and Building Convolutional Neural Networks using PyTorch. Models, selected are based on number of ci

Mayur 126 Dec 22, 2022
MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images This repository contains the implementation of our paper MetaAvatar: Learni

sfwang 96 Dec 13, 2022
68 keypoint annotations for COFW test data

68 keypoint annotations for COFW test data This repository contains manually annotated 68 keypoints for COFW test data (original annotation of CFOW da

31 Dec 06, 2022
LSUN Dataset Documentation and Demo Code

LSUN Please check LSUN webpage for more information about the dataset. Data Release All the images in one category are stored in one lmdb database fil

Fisher Yu 426 Jan 02, 2023
Probabilistic Programming and Statistical Inference in PyTorch

PtStat Probabilistic Programming and Statistical Inference in PyTorch. Introduction This project is being developed during my time at Cogent Labs. The

Stefano Peluchetti 109 Nov 26, 2022
Source code for ZePHyR: Zero-shot Pose Hypothesis Rating @ ICRA 2021

ZePHyR: Zero-shot Pose Hypothesis Rating ZePHyR is a zero-shot 6D object pose estimation pipeline. The core is a learned scoring function that compare

R-Pad - Robots Perceiving and Doing 18 Aug 22, 2022
Spam your friends and famly and when you do your famly will disown you and you will have no friends.

SpamBot9000 Spam your friends and family and when you do your family will disown you and you will have no friends. Terms of Use Disclaimer: Please onl

DJ15 0 Jun 09, 2022
Tutorial on scikit-learn and IPython for parallel machine learning

Parallel Machine Learning with scikit-learn and IPython Video recording of this tutorial given at PyCon in 2013. The tutorial material has been rearra

Olivier Grisel 1.6k Dec 26, 2022