Python package for downloading ECMWF reanalysis data and converting it into a time series format.

Overview

ecmwf_models

https://github.com/TUW-GEO/ecmwf_models/workflows/Automated%20Tests/badge.svg?branch=master https://coveralls.io/repos/github/TUW-GEO/ecmwf_models/badge.svg?branch=master https://readthedocs.org/projects/ecmwf-models/badge/?version=latest

Readers and converters for data from the ECMWF reanalysis models. Written in Python.

Works great in combination with pytesmo.

Citation

If you use the software in a publication then please cite it using the Zenodo DOI. Be aware that this badge links to the latest package version.

Please select your specific version at https://doi.org/10.5281/zenodo.593533 to get the DOI of that version. You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility.

You can find additional information regarding DOI versioning at http://help.zenodo.org/#versioning

Installation

Install required C-libraries via conda. For installation we recommend Miniconda. So please install it according to the official installation instructions. As soon as you have the conda command in your shell you can continue:

conda install -c conda-forge pandas pygrib netcdf4 pyresample xarray

The following command will download and install all the needed pip packages as well as the ecmwf-model package itself.

pip install ecmwf_models

To create a full development environment with conda, the yml files inside the folder environment/ in this repository can be used. Both environements should work. The file latest should install the newest version of most dependencies. The file pinned is a fallback option and should always work.

git clone --recursive [email protected]:TUW-GEO/ecmwf_models.git ecmwf_models
cd ecmwf_models
conda env create -f environment/latest.yml
source activate ecmwf_models
python setup.py develop
pytest

Supported Products

At the moment this package supports

  • ERA Interim (deprecated)
  • ERA5
  • ERA5-Land

reanalysis data in grib and netcdf format (download, reading, time series creation) with a default spatial sampling of 0.75 degrees (ERA Interim), 0.25 degrees (ERA5), resp. 0.1 degrees (ERA5-Land). It should be easy to extend the package to support other ECMWF reanalysis products. This will be done as need arises.

Contribute

We are happy if you want to contribute. Please raise an issue explaining what is missing or if you find a bug. Please take a look at the developers guide.

Comments
  • Fix returned status code in case of partial data availability

    Fix returned status code in case of partial data availability

    Consider the case of downloading data for 6 months. This script does it month by month. If data does not exist for the last month the return code was -10 before this fix, indicating that no data at all is available.

    Now we only get a -10 return code if indeed no data at all is available in the requested period

    opened by cpaulik 5
  • Handle downloads no data available

    Handle downloads no data available

    Fixes https://github.com/TUW-GEO/ecmwf_models/issues/26

    If the CDS API does not have any data available we catch this with the error callback. Only the string Reason: Request returned no data is available to catch this. If the CDS package would ever change their error messages this would break.

    As a return code the python errno.ENODATA is used if no data is present. I've not been able to find more widely used default error codes so I would argue that also just using -10 by convention would be ok instead.

    Other changes

    I've also removed the usage of the * import in this MR.

    opened by cpaulik 4
  • Handle requests that fail because no data is available yet

    Handle requests that fail because no data is available yet

    Downloading e.g. era5-land data of the last month gives an error like the following

    era5_download /home/cpa/debugging/2021-10-20_era5_download/era5_land/ -s 2021-10-01 -e 2021-10-10 -p era5-land -var "volumetric_soil_water_layer_1" --h_steps 2
    Downloading era5-land netcdf files between 2021-10-01T00:00:00 and 2021-10-10T00:00:00 into folder /home/cpa/debugging/2021-10-20_era5_download/era5_land/
    2021-10-22 13:00:31,336 INFO Welcome to the CDS
    2021-10-22 13:00:31,337 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-land
    2021-10-22 13:00:31,453 INFO Request is queued
    2021-10-22 13:12:55,443 INFO Request is failed
    2021-10-22 13:12:55,444 ERROR Message: no data is available within your requested subset
    2021-10-22 13:12:55,444 ERROR Reason:  Request returned no data
    2021-10-22 13:12:55,445 ERROR   Traceback (most recent call last):
    2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
    2021-10-22 13:12:55,445 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
    2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
    2021-10-22 13:12:55,445 ERROR       result = proc(context, *context.args, **context.kwargs)
    2021-10-22 13:12:55,445 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
    2021-10-22 13:12:55,446 ERROR       return p(*args, **kwargs)
    2021-10-22 13:12:55,446 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
    2021-10-22 13:12:55,446 ERROR       return self.proc(context, *args, **kwargs)
    2021-10-22 13:12:55,446 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 45, in internal
    2021-10-22 13:12:55,446 ERROR       return mars(context, request, **kwargs)
    2021-10-22 13:12:55,446 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 17, in mars
    2021-10-22 13:12:55,447 ERROR       execute_mars(context, requests)
    2021-10-22 13:12:55,447 ERROR     File "/home/cds/cdsservices/services/mars/execute_mars.py", line 25, in execute_mars
    2021-10-22 13:12:55,447 ERROR       raise NoDataException("Request returned no data", '')
    2021-10-22 13:12:55,447 ERROR   cdsinf.exceptions.NoDataException: Request returned no data
    

    In the current implementation that means that the downloading is retried 5 times after which the program fails when trying to open the downloaded NetCDF or GRIB file.

    Ideally the CLI would show a error message and return with a negative exit code

    I think we could catch that by using an error_callback that cdsapi provides. But maybe another solution will be necessary.

    opened by cpaulik 2
  • pygrib 1.9.9

    pygrib 1.9.9

    Hey! I was just fixing some CI issues. All python versions are passing now, but I saw that there are CI environments for pygrib version 1.9.9 and version 2.0.1. Or at least there should be, because until now only version 2.0.1 was tested (which was installed last from the requirements.txt via pip install -r requirements.txt in .travis.yml).

    Now I have the 2 versions working, and tests fail for version 1.9.9 as some of the metadata stuff is handled differently it seems. e.g grbs.message(1)['levels'] returnsu'0-7' for v2.0.1 but u'7' for v1.9.9 (on the same file)

    Should we support and test v1.9.9? Otherwise I would just set pygrib>=2.0.1 as the required version if thats ok for you?

    opened by wpreimes 2
  • ERA 5

    ERA 5

    I think supporting ERA 5 download and ts generation would be a good idea (so that it works when the full data set is available in a few months). https://www.ecmwf.int/en/newsletter/147/news/era5-reanalysis-production

    opened by wpreimes 2
  • Restructure and new ERA5 API

    Restructure and new ERA5 API

    Tried to separate ERA Interim and ERA5 more, so that we can focus on developing the ERA5 part better in the future. era5_download uses the cds api (instead of ecmwf api). There are more options for downloading now, and we can add more if we want, also the download functions are tested now.

    opened by wpreimes 1
  • new download api

    new download api

    Looks like download of era5 from february on has to be done via a new api.

    "Access through ECMWF Web-API to dataset ERA5 and C3S Seasonal will be phased out, please migrate to the Climate Data Store (https://cds.climate.copernicus.eu/)."

    https://confluence.ecmwf.int/display/CKB/C3S+ERA5%3A+Web+API+to+CDS+API

    opened by wpreimes 1
  • Era5 and netcdf support

    Era5 and netcdf support

    Can we release this as a new version? I think there were some significant changes:

    • Add ERA5 support (download, reading, reshuffling)
    • Add netcdf support for ERA5 and ERA-Interim download (regular grid)
    • Netcdf download in regular grid, grib in gaussian grid -- download allows passing parameters for spatial resampling now (grib and nc)
    • Rework GRIB message storing (dont create a new file for each message, but save messages for each day in a distinct file)
    • Add more tests

    We should consider adding:

    • Grid reducion to land mask when creating TS (Land mask is already being downloaded automatically)
    • Lookup table for parameter ids and variable names (download now via parameter ids, reshuffling via variable names necessary as param ids not saved when downloading netcdf files)
    opened by wpreimes 1
  • A few more changes for downloading hourly data and regridding with CDO

    A few more changes for downloading hourly data and regridding with CDO

    • Option for a different stepsize than full months for downloads. For hourly data, the requests are too large otherwise
    • Removal of temporary files created by CDO after regridding, otherwise the temporary directory will be filled up when using jupyter notebooks.
    opened by s-scherrer 0
  • Regridding with CDO

    Regridding with CDO

    This adds the option to regrid data directly after downloading it using CDO. The regridding is done using pre-computed weights in a separate thread in order to not block the download.

    opened by s-scherrer 0
  • Fix grib and era5-land grid

    Fix grib and era5-land grid

    • Fix bug when creating 0.1 deg grid cells (floating point precision)
    • Missing variables in grib files are now replaced by empty images.
    • Read variable names from grib files from cfVarNameECMF instead of short_name field
    opened by wpreimes 0
Releases(v0.9.1)
  • v0.9.1(May 3, 2022)

  • v0.9.0(Mar 18, 2022)

  • v0.8.1(Feb 14, 2022)

  • v0.8(Nov 15, 2021)

    • Program era5_download returns exit code now (PR #27);
    • Program era5_reshuffle can now take a bounding box to reshuffle spatial subsets;
    • TravisCI was replaced by Github Actions;
    • Pyscaffold 4 is used; contributing guide added; pre-commit added;
    • Code formatting with black (line length 79);
    Source code(tar.gz)
    Source code(zip)
  • v0.7(Jun 19, 2020)

  • v0.6.1(Jan 9, 2020)

    • Fix bug when creating 0.1 deg grid cells (floating point precision)
    • Missing variables in grib files are now replaced by empty images.
    • Read variable names from grib files from cfVarNameECMF instead of short_name field
    Source code(tar.gz)
    Source code(zip)
  • v0.6(Sep 20, 2019)

    • Add support for downloading, reading, reshuffling era5-land
    • Add support for reading, reshuffling points over land only (era5 and era5-land)
    • Add function to create land definition files
    • Test with pinned environments
    Source code(tar.gz)
    Source code(zip)
  • v0.5(Jun 13, 2019)

    • Change default time steps to 6 hours.
    • Add more tests, also for download functions
    • Update documentation, add installation script
    • Fix bugs, update command line interfaces, update dependencies
    • Separate download programs for ERA5 and ERA Interim
    • Change the ERA5 download api to use cdsapi instead of ecmwf api
    • Update package structure to better separate between the ERA products
    • Add look-up-table file for more flexibility in variable names passed by user
    • Update readme
    Source code(tar.gz)
    Source code(zip)
  • v0.4(Dec 15, 2018)

    • Add ERA5 support (download, reading, TS conversion)
    • Add netcdf support for ERA5 and ERA-Interim download (regular grid)
    • Add new grid defintions: netcdf download in regular grid, grib in gaussian grid
    • Add Download with spatial resampling (grib and nc)
    • Update GRIB message storing (per day instead of per message)
    Source code(tar.gz)
    Source code(zip)
  • v0.3(Jan 20, 2017)

    • Fix help text in ecmwf_repurpose command line program.
    • Fix reading of metadata for variables that do not have 'levels'
    • Fix wrong import when trying to read the reformatted time series data.
    Source code(tar.gz)
    Source code(zip)
  • v0.2(Sep 5, 2016)

    • Add reading of basic metadata fields name, depth and units.
    • Fix reading of latitudes and longitudes - where flipped before.
    • Fix longitude range to -180, 180.
    • Add conversion to time series format.
    Source code(tar.gz)
    Source code(zip)
  • v0.1(Aug 30, 2016)

Owner
TU Wien - Department of Geodesy and Geoinformation
TU Wien - Department of Geodesy and Geoinformation
This repository implements WGAN_GP.

Image_WGAN_GP This repository implements WGAN_GP. Image_WGAN_GP This repository uses wgan to generate mnist and fashionmnist pictures. Firstly, you ca

Lieon 6 Dec 10, 2021
Open-source implementation of Google Vizier for hyper parameters tuning

Advisor Introduction Advisor is the hyper parameters tuning system for black box optimization. It is the open-source implementation of Google Vizier w

tobe 1.5k Jan 04, 2023
ConvMAE: Masked Convolution Meets Masked Autoencoders

ConvMAE ConvMAE: Masked Convolution Meets Masked Autoencoders Peng Gao1, Teli Ma1, Hongsheng Li2, Jifeng Dai3, Yu Qiao1, 1 Shanghai AI Laboratory, 2 M

Alpha VL Team of Shanghai AI Lab 345 Jan 08, 2023
Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"

ON-LSTM This repository contains the code used for word-level language model and unsupervised parsing experiments in Ordered Neurons: Integrating Tree

Yikang Shen 572 Nov 21, 2022
This repo contains the code and data used in the paper "Wizard of Search Engine: Access to Information Through Conversations with Search Engines"

Wizard of Search Engine: Access to Information Through Conversations with Search Engines by Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zh

19 Oct 27, 2022
MANO hand model porting for the GraspIt simulator

Learning Joint Reconstruction of Hands and Manipulated Objects - ManoGrasp Porting the MANO hand model to GraspIt! simulator Yana Hasson, Gül Varol, D

Lucas Wohlhart 10 Feb 08, 2022
The code of "Dependency Learning for Legal Judgment Prediction with a Unified Text-to-Text Transformer".

Code data_preprocess.py: preprocess data for Dependent-T5. parameters.py: define parameters of Dependent-T5. train_tools.py: traning and evaluation co

1 Apr 21, 2022
Learning To Have An Ear For Face Super-Resolution

Learning To Have An Ear For Face Super-Resolution [Project Page] This repository contains demo code of our CVPR2020 paper. Training and evaluation on

50 Nov 16, 2022
A PyTorch implementation for V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

A PyTorch implementation of V-Net Vnet is a PyTorch implementation of the paper V-Net: Fully Convolutional Neural Networks for Volumetric Medical Imag

Matthew Macy 606 Dec 21, 2022
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 3.7k Jan 03, 2023
Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021. Introduction We proposed a novel model training paradi

Lucas 103 Dec 14, 2022
Reimplementation of NeurIPS'19: "Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting" by Shu et al.

[Re] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting Reimplementation of NeurIPS'19: "Meta-Weight-Net: Learning an Explicit Mapping

Robert Cedergren 1 Mar 13, 2020
OpenDILab RL Kubernetes Custom Resource and Operator Lib

DI Orchestrator DI Orchestrator is designed to manage DI (Decision Intelligence) jobs using Kubernetes Custom Resource and Operator. Prerequisites A w

OpenDILab 205 Dec 29, 2022
[Machine Learning Engineer Basic Guide] 부스트캠프 AI Tech - Product Serving 자료

Boostcamp-AI-Tech-Product-Serving 부스트캠프 AI Tech - Product Serving 자료 Repository 구조 part1(MLOps 개론, Model Serving, 머신러닝 프로젝트 라이프 사이클은 별도의 코드가 없으며, part

Sung Yun Byeon 269 Dec 21, 2022
Pytorch-3dunet - 3D U-Net model for volumetric semantic segmentation written in pytorch

pytorch-3dunet PyTorch implementation 3D U-Net and its variants: Standard 3D U-Net based on 3D U-Net: Learning Dense Volumetric Segmentation from Spar

Adrian Wolny 1.3k Dec 28, 2022
这是一个yolo3-tf2的源码,可以用于训练自己的模型。

YOLOV3:You Only Look Once目标检测模型在Tensorflow2当中的实现 目录 性能情况 Performance 所需环境 Environment 文件下载 Download 训练步骤 How2train 预测步骤 How2predict 评估步骤 How2eval 参考资料

Bubbliiiing 68 Dec 21, 2022
Res2Net for Instance segmentation and Object detection using MaskRCNN

Res2Net for Instance segmentation and Object detection using MaskRCNN Since the MaskRCNN-benchmark of facebook is deprecated, we suggest to use our mm

Res2Net Applications 55 Oct 30, 2022
Incomplete easy-to-use math solver and PDF generator.

Math Expert Let me do your work Preview preview.mp4 Introduction Math Expert is our (@salastro, @younis-tarek, @marawn-mogeb) math high school graduat

SalahDin Ahmed 22 Jul 11, 2022
Official code for 'Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning' [ICCV 2021]

RTFM This repo contains the Pytorch implementation of our paper: Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Lear

Yu Tian 242 Jan 08, 2023
Lite-HRNet: A Lightweight High-Resolution Network

LiteHRNet Benchmark 🔥 🔥 Based on MMsegmentation 🔥 🔥 Cityscapes FCN resize concat config mIoU last mAcc last eval last mIoU best mAcc best eval bes

16 Dec 12, 2022