Data in HDF file formats continues to play an important role for Earth Scientists in the U.S. and around the world. The HDF Group will update ESIP members on the state of HDF software and HDF5 Roadmap, and will share our experience on working with HDF5 in the Cloud. We will discuss our technical approaches, and lessons learned from different projects including a NASA ACCESS project that transformed NASA HDF data into GeoTIFF in AWS. We will also update ESIP members on our involvement in standardization efforts and demonstrate how HDF tools support ESDIS data from product initial design to production, and to compliance with the standards. We will encourage ESIP members participating in the session to share their experiences with the HDF software and to contribute to the HDF5 Roadmap.
Talks
Google Colaboratory for HDF-EOS - Joe Lee Abstract: Google provides a free Jupyter notebook environment called Colaboratory (also known as Colab). It is simple, easy, and awesome Python environment for data scientists. We present how NASA Earthdata in HDF can be used with Google Colab using the existing comprehensive example on HDF-EOS Tools and Information Center website (http://hdfeos.org/zoo). We also present how OPeNDAP can be used with Colab to achieve 100%-cloud data analysis.
Keywords: Python, Google Colab, Jupyter notebook, HDF-EOS, OPeNDAP, Cloud computing.
Slides: https://doi.org/10.6084/m9.figshare.8976464Leveraging the Cloud for HDF Software Testing - Larry Knox
Abstract: In this talk we will discuss how we leverage the Cloud for HDF software daily regression testing including testing of the HDF5 parallel library on the Cloud cluster using Orange FS.Keywords: HDF5, Cloud, CI testing.
Parallel Computing with HDF Server - John Readey
Abstarct:
To deal with really big data you need to be able to harness the power of multiple machines, but many users are put off by the complexity involved in setting up a cluster and then figuring out to effectively utilize it. However, by using HDF Server (HSDS) with Kubernetes, it’s much easier than you would think. In this talk we’ll walk through some examples of using xarray, h5netcdf, and h5py with HSDS to illustrate how you can scale up your compute to match your data size.Keywords: HDF5, h5netcdf, h5py
HDF5 Roadmap 2019-2020 - Elena Pourmal
Abstract: In this talk we will give an overview of the new features of the upcoming HDF5 release 1.12.0, and outline the HDF5 roadmap for the next year. We will demonstrate new open source file drivers to access HDF5 files via Amazon Simple Storage Service (Amazon S3) and on Hadoop Distributed File system (HDFS). We will use this presentation to get feedback on the HDF5 roadmap from the ESDIS users and application developers.Keywords: HDF5, Amazon S3, HDFS, Cloud, Object Store.
Session recording here.
