2019 ESIP Summer Meeting

The Information Quality Cluster (IQC) has been active since 2014 improving understanding of various aspects of information quality and fostering collaborations nationally and internationally. During this period, NASA’s Earth Science Data System Working Groups included a Data Quality Working Group, which made several recommendations that have been documented, reviewed thoroughly and published. The IQC has had plenary and breakout sessions discussing ideas about uncertainty in Earth science datasets, which have evolved into a white paper. Significant progress has been made in defining and propagating maturity matrices for various aspects of data management including information quality. The purpose of this session is to summarize the status and accomplishments in each of these areas and discuss future directions that the IQC should take.

Agenda
1. Information Quality Cluster Introduction - H. K. Ramapriyan (Rama) - 10 mins
2. NASA Data Quality Working Group’s Recommendations and Publications – Yaxing Wei – 20 mins.
3. Uncertainty White Paper Status – David Moroni – 15 mins.
4. Data Quality @Open Geospatial Consortium – Ivana Ivanova - 15 mins.
5. Maturity Matrices Update – Ge Peng – 15 mins.
6. Discussion and Key Takeaways – All – 15 mins.

View All Slides: https://doi.org/10.6084/m9.figshare.9336248

Meeting Notes: http://bit.ly/IQC_20190716_Notes  (for group editing)

View Session Recording on YouTube.

Session Take-Aways

1. Considerable progress has been made in several areas over the last 5 years. Progress reported from NASA’s Data Quality Working Group, NOAA’s Data Stewardship Maturity Matrices, OGC’s Data Quality Domain Working Group. Collaborations have developed during the last two years with connections established with other clusters and non-US/international groups including E2SIP. 
2. Members and observers of the IQC are looking to the IQC chairs/co-chairs for new opportunities to engage the community outside of ESIP at external conferences, namely AGU. Community engagement efforts to seek outside collaboration at IQC-organized Fall AGU sessions have been quite successful, particularly in recruiting presenters for invited talks in monthly telecons and collaborators for the uncertainty white paper.
3. Progress is being made on development of a white paper on Earth science data uncertainty. IQC should consider airborne and in situ data as well - some new use cases are needed.
   
   

In keeping with this year’s Summer Meeting theme of “Increasing the Use and Value of Earth Science Data and Information,” this session aims to explore different data streams used for monitoring of the hydrosphere and cryosphere. Earth science data for water resources monitoring has existed as field collected data, remote sensing, modeled and in situ data for decades but relatively recent increases in computational capabilities (e.g. cloud computing platforms), data storage and integration and processing methods like machine learning have allowed researchers to ask a suite of questions that rely on data from multiple sources and typologies to answer complex questions about water resources critical to humans and ecosystems. To emphasize the ‘use and value of earth science data’ this session will incorporate presentations on data generation and processing methods as well as applied uses of data products for water resources monitoring.

Presenter: Eric Sproles
Presentation Title: Bridging the Scaling Issues of Earth Observations
Slides: https://doi.org/10.6084/m9.figshare.8980400

Presenter: Jeffrey Deems
Presentation Title: New Data, Old Problems: Integrating Novel Data Sources for Study & Management of Snowmelt Systems
Slides: https://doi.org/10.6084/m9.figshare.8980406

Presenter: Yuhan Rao
Presentation Title: Integrating Satellite Observations and In Situ Measurements to Study Snow-Albedo-Temperature Interactions Over the Tibetan Plateau
Slides: https://doi.org/10.6084/m9.figshare.8980409

Presenter: Scott Oviatt
Presentation Title: National Resources Conservation Service SNOTEL Network
Slides: https://doi.org/10.6084/m9.figshare.8980415

Presenter: Ruth Duerr
Presentation Title: Polar Data Activities
Slides: https://doi.org/10.6084/m9.figshare.8980397

Session Take-Aways

1. NRCS plans to convert long-term snow courses to SNOTEL, continue to pursue tech upgrades, develop new methodologies to improve accuracy
2. Machine learning can integrate satellite observations and in situ measurements to create a more complete measurement
3. UAV provide higher density albedo measurements, remote locations, multiple field sites
4. Creating an integrated system for the future to track cryospheric changes
5. Arctic Data Committee has technical and semantic guidance for integrating cryospheric data

View the Recording on YouTube

Data in HDF file formats continues to play an important role for Earth Scientists in the U.S. and around the world. The HDF Group will update ESIP members on the state of HDF software and HDF5 Roadmap, and will share our experience on working with HDF5 in the Cloud. We will discuss our technical approaches, and lessons learned from different projects including a NASA ACCESS project that transformed NASA HDF data into GeoTIFF in AWS. We will also update ESIP members on our involvement in standardization efforts and demonstrate how HDF tools support ESDIS data from product initial design to production, and to compliance with the standards. We will encourage ESIP members participating in the session to share their experiences with the HDF software and to contribute to the HDF5 Roadmap.

Talks   
Google Colaboratory for HDF-EOS - Joe Lee Abstract: Google provides a free Jupyter notebook environment called Colaboratory (also known as Colab).  It is simple, easy, and awesome Python environment for data scientists. We present how NASA Earthdata in HDF can be used with Google Colab using the existing comprehensive example on HDF-EOS Tools and Information Center website (http://hdfeos.org/zoo). We also present how OPeNDAP can be used with Colab to achieve 100%-cloud data analysis.

Keywords: Python, Google Colab, Jupyter notebook, HDF-EOS, OPeNDAP, Cloud computing.
Slides: https://doi.org/10.6084/m9.figshare.8976464

Leveraging the Cloud for HDF Software Testing - Larry Knox

Abstract: In this talk we will discuss how we leverage the Cloud for HDF software daily regression testing including testing of the HDF5 parallel library on the Cloud cluster using Orange FS.
Keywords: HDF5, Cloud, CI testing.
Parallel Computing with HDF Server - John Readey

Abstarct: To deal with really big data you need to be able to harness the power of multiple machines, but many users are put off by the complexity involved in setting up a cluster and then figuring out to effectively utilize it.   However, by using HDF Server (HSDS) with Kubernetes, it’s much easier than you would think.  In this talk we’ll walk through some examples of using xarray, h5netcdf, and h5py with HSDS to illustrate how you can scale up your compute to match your data size.
Keywords: HDF5, h5netcdf, h5py

 HDF5 Roadmap 2019-2020 - Elena Pourmal

Abstract: In this talk we will give an overview of the new features of the upcoming HDF5 release 1.12.0, and outline the HDF5 roadmap for the next year. We will demonstrate new open source file drivers to access HDF5 files via Amazon Simple Storage Service (Amazon S3) and on Hadoop Distributed File system (HDFS). We will use this presentation to get feedback on the HDF5 roadmap from the ESDIS users and application developers.
Keywords: HDF5, Amazon S3, HDFS, Cloud, Object Store.

Session recording here.

We know all the standard mechanisms for integrating data systems with each other: standards, APIs, standards-based APIs, etc., etc., etc. But new possibilities are opening up due to new technologies and approaches: Jupyter, Eclipse Che, Everything-as-a-Service, Slack, JSON-LD... Do you have a novel integration mechanism you want more developers to adopt so we can hook more things together? Come to this session and talk it up!

Agenda:

• Dave Blodgett (USGS) - "Non information resource - Meta information resource - Data information resources: A resource model for integration of data from multiple organizations about the same real-world feature. Summary of the ongoing the Second Environmental Linked Features Interoperability Experiment (SELFIE)"
• Kevin O'Brien/Bob Simons (NOAA/PMEL) - Integrating Data with ERDDAP
• Namrata Malarout (NASA/JPL) - Centralized MAAP* API: Simplifying Algorithm Collaboration
• Daven Quinn (U. Wisconsin) - Sparrow: an in-house, interoperable data system for individual geochronology labs

*MAAP = Multi-mission Algorithm and Analysis Platform

Session recording here.

Earth observation (EO) satellite files can encode the geolocation of their
observations in a variety of ways. This often depends on the processing
level of information. Gridded rasters can be described by concise map
projection information while Level 1 and 2 data that are in native satellite
coordinates require more detail. Often these files encode the pixel level
geolocation information as multi dimension variables internal to the file.
In the past there have been example implementations of storing geolocation
in an external file (early NASA MODIS) or sub sampling geolocation
information (early NASA SeaWiFS) that did not work out very well for
various reasons. Storing the geolocation data or map projection references
in each file (granule) has many advantages the most important is playing
"nicely" with tools and services and software, and promoting
interoperability. However, the geolocation data for L1/L2 EO files are
often the storage heaviest individual component as its precision requires at
least float data types (its information cannot be elegantly "packed") so it
is worthwhile to revisit ideas and methodologies for reducing its
footprint. How best could geolocation information be shared across
different variables, different files from the same sensor, or even
different sensors on the same platform. Furthermore, in the age of cloud
and database tiled storage of satellite information how is geo-location (and
other) information best packaged and utilized to improve data access and
processing. In this session we will look at this problem and potential
solution space via a number of presentations, historical lessons learned and
dynamic discussion.


Presentations:

• Introduction: Ed Armstrong and Alexsander Jelenak
• Kwo-Sen Kuo: STARE and data packaging
• Robert Wolf: MODIS Experience with External Geolocation
• Ed Armstrong; Reducing geolocation storage: a CF success story, and introducting a novel (and complicated) Eumetsat L2 data model 
• Alexsandar Jelenak: HDF5 and external references

Find and access all slides: https://doi.org/10.6084/m9.figshare.8986325

Session recording here.