2019 ESIP Summer Meeting

Increasing user adoption of and applications for cloud technologies as well as exponential growth in data volumes demands our public sector data archives accommodate cloud computing. Simultaneously, government-funded computing environments have constraints that present unique challenges in providing archives in the cloud, including Trusted Internet Connection mandates, funding models and legislation which do not allow unbounded costs, and security policies inherited from a pre-cloud world. Join us to discuss the progress members of the ESIP community have made in overcoming these hurdles toward moving large public sector archives to the cloud for valuable science applications.

View Full Recording on YouTube

Presenter: Nathan Clark
Talk Title: Cost Controls in the Cloud
Slides: https://doi.org/10.6084/m9.figshare.8938514

Presenter: Ben Williams
Talk Title: Cloud Governance at Scale
Slides: https://doi.org/10.6084/m9.figshare.8939981

Are you new to ESIP? Join us for a quick primer on ESIP. 

When data is shared in the cloud, anyone can analyze it without having to download it or store it themselves, which lowers the cost of new product development, reduces the time to scientific discovery, and can accelerate innovation. However, staging large-scale datasets for analysis in the cloud requires consideration of how data should be prepared and organized to allow fast, efficient, and programmatic access from distributed computing systems. This workshop provides a forum for members of the community to share lessons learned as they explore ways to use the cloud to expand data access. It seeks to encourage dialog between users interested in leveraging data in the AWS Cloud for research and application development for Earth Sciences.

View Session Recording

Session Description:
When data is shared in the cloud, anyone can analyze it without having to download it or store it themselves, which lowers the cost of new product development, reduces the time to scientific discovery, and can accelerate innovation. However, staging large-scale datasets for analysis in the cloud requires consideration of how data should be prepared and organized to allow fast, efficient, and programmatic access from distributed computing systems. This workshop provides a forum for members of the community to share lessons learned as they explore ways to use the cloud to expand data access. It seeks to encourage dialog between users interested in leveraging data in the AWS Cloud for research and application development for Earth Sciences.

Workshop Format: 
Workshop includes 1.5 hours of presentations (Cloud Data Optimization: Emerging Best Practices I) followed by 1.5 hours of discussion on emerging best practices and identifying needs to move this space forward.

Presentations (10 minutes each)
Full Abstracts can be found in the attached file.

1. Title: STAC, sat-utils, and Open Data - Prioritizing Data Use (10 min)
   Presenter: Dan Pilone (Element 84)
2. Title: Radiant ML Hub, A cloud based commons for geospatial training datasets (10 min)
   Presenter: Hamed Alemohammad (Radiant Earth Foundation)
   Slides: https://doi.org/10.6084/m9.figshare.9696446
3. Title: One data format pattern to rule them all (10 min)
   Presenter: Grega Milcinski (Sinergise)
   Slides: https://doi.org/10.6084/m9.figshare.9121991
4. Title: Improved Cloud Raster Format for multidimensional raster storage and analysis (10 min)
   Presenters: Hong Xu (Esri) & Sudhir Raj Shrestha (Esri)
   Slides: https://doi.org/10.6084/m9.figshare.9762866
5. Title: Optimization of CESM LENS on AWS S3 (10 min)
   Presenter: Jeff de La Beaujardiere (NCAR)
   Slides: https://doi.org/10.6084/m9.figshare.9633314
6. Title: The Zarr format
   Presenter: Rich Signell (USGS)
   Slides: https://doi.org/10.6084/m9.figshare.9701684
7. Title: NOAA’s Big Data Project - A Data Broker’s Perspective
   Presenter: Otis Brown (NC State University/NCICS)
   Slides: https://doi.org/10.6084/m9.figshare.9693776
8. Title: HDF Data Service for the Cloud
   Presenter: John Readey (The HDF Group)

Session Take-Aways

1. Moving to cloud infrastructure offers a chance to reevaluate best practices, though some of these may not be purely cloud-related (e.g., data formats) but the discussions are coming along for the ride!
2. It is unclear who will own the cloud-optimized datasets and it will likely be different from dataset to dataset. Until (if/when) cloud-optimized formats become the norm, they may often be provided by other groups (or created on the fly).
3. There is a lot of focus on datasets in these conversations, but we need to also focus on tooling/services and education.

Event Tittle: Cloud Data Optimization: Emerging Best Practuces II
Event Date/Time: July 16 | 4:30 PM-6:00 PM
Event Location: Greater Tacoma Convention Center, Tacoma, WA
ESIP URL: esipfed.org/summermeeting
Session Moderators: Ana Pinheiro Privette (Amazon), Joe Flasher (AWS) and Jeff de La Beaujardiere (NCAR)

View Session Recording on YouTube.

Session Description:
When data is shared in the cloud, anyone can analyze it without having to download it or store it themselves, which lowers the cost of new product development, reduces the time to scientific discovery, and can accelerate innovation. However, staging large-scale datasets for analysis in the cloud requires consideration of how data should be prepared and organized to allow fast, efficient, and programmatic access from distributed computing systems. This workshop provides a forum for members of the community to share lessons learned as they explore ways to use the cloud to expand data access. It seeks to encourage dialog between users interested in leveraging data in the AWS Cloud for research and application development for Earth Sciences.

Workshop Format: 
Workshop includes 1.5 hours of presentations (Cloud Data Optimization: Emerging Best Practices I) followed by 1.5 hours of discussion on emerging best practices and identifying needs to move this space forward.

Session II - Round table discussion: emerging best practices in cloud data optimization

PROBLEM: schema.org can describe static Datasets, but it's difficult to accurately describe services and APIs that provide access to data. This session will bring together data API managers and curators, conceptual modelers and ontologists to model and develop a schema.org extension address accessing data through APIs and services.

Session recording is here.

PROBLEM: schema.org is currently inconsistent with standards organizations (W3C, OGC) representations of spatial and temporal information. This session will bring together data curators, conceptual modelers and ontologists to formulate solutions for extending schema.org's approach to spatial and temporal descriptions.

Session recording here.

Repositories have recognized the benefits of adopting schema.org metadata in their data catalog landing pages to improve discoverability, particularly with the incentive of inclusion in the Google Dataset search. While Google supports broad, general search and discovery, we can also use this mechanism to improve domain-specific aggregated search systems like DataONE. In this working session, we will focus on real world issues of implementing schema.org for repositories, how to link traditional metadata records into dataset landing pages, and how this can result in improved harvesting and representation by science focused aggregators such as DataONE. We will work through recommendations emerging from science-on-schema.org, optimizing JSON-LD to work with major search engines, and options for extending to include more detailed dataset information beyond the typical discovery-level metadata found in most records.

Session recording here.

Because they care about and for the infrastructure that houses every bit of data, every byte of the cloud, and every line of code, maintainers sustain the technology infrastructure that makes Earth data use possible. Maintainers work in many arenas, of course, they keep energy grids up, roadways repaired, buildings secure. Data infrastructure experts are now in conversations with other maintainers. Recently, a group of maintainers: technicians, engineers, historians, social scientists, sysadmins (the ones you call on to reboot the system when it’s down) started a conversation and created a group called The Maintainers. With support from the Alfred P. Sloan Foundation, ESIP is bringing the Maintainer conversation to Tacoma. We’ve invited several of them to talk about the real issues involved in stewarding hardware and systems, not just data. By caring for your hardware, they let you focus on other tasks. Join us to discover how ESIP’s goals of sustaining the Earth science data endeavor rely upon those who chose not to innovate today, but rather to navigate the problematics of keeping everything running most of the time.

Session recording here.

Introduction: Bruce Caron
Presentation Title: Culture, Kindness, and Care: Commoning for Earth Knowledge Sustainability
Slides: https://doi.org/10.6084/m9.figshare.8969912

Moderator: Mark Parsons
Slides: https://doi.org/10.6084/m9.figshare.8969915

Invited Presentations
Presenter: Emily Jane Sylak-Glassman
Presentation Title: The Importance of Maintaining Earth Observational Data for Long-Term Climate Record Reconstruction
Slides: https://doi.org/10.6084/m9.figshare.8969918

Presenter: Daniella Lowenberg
Presentation Title: Maintaining and Growing Research Data Publishing at CDL & Dryad
Slides: https://doi.org/10.6084/m9.figshare.8980454

Presenter: Jason A. Gallo
Presentation Title: The Scale and Value of Earth Observation Infrastructure
Slides: https://doi.org/10.6084/m9.figshare.8969924

Presenter: Fred C. Beach
Presentation Title: U.S. Energy Infrastructure: ‘What’s Past is Prologue’
Slides: https://doi.org/10.6084/m9.figshare.8969921

Session Take-Aways

1. ESIP scientists and data scientists and project managers are only one part of the larger team that keeps Earth information active and durable. The Maintainer organization can be an “ESIP for the rest of the team”... where ESIP maintainers gather with others to solve their problems. ESIP will have a session at the next Information Maintainer conference in DC in October. Maintenance is about fostering and caring for relationships: Who decides to maintain is all of us. This requires awareness and kindness.
2. Earth data resources often have multiple inputs, some of these quite complex. Just finding out and mapping these is an important maintainer activity. Also, archiving data and software at the same time makes good sense (Dryad and Zenodo). Maintenance is more complex than it seems (The US has no energy policy; predicting one variable (sea ice extent) requires dozens of inputs and complex interactions).
3. Infrastructure (such as our energy infrastructure) can get to the point where the trillions of dollars needed to update it might be better spent replacing this with something highly distributed.

"Brains! Brains! Give us your brains!"               
      - Friendly neighbourhood machine minds
The collective knowledge in the ESIP community is immense and invaluable. During this session, we'd like to make sure that this knowledge drives the semantic technology (ontologies) being developed to move data with machine-readable knowledge in Earth and planetary science.
What we'll do:

1. In the first half hour of this session, we'll a) sketch out how and why we build ontologies and b) show you how to request that your knowledge gets added to ontologies (with nanocrediting).
2. We'll then have a 30-minute crowdsourcing jam session, during which participants can share their geoscience knowledge on the SWEET issue tracker. With a simple post, you can shape how the semantic layer will behave, making sure it does your field justice! Request content and share knowledge here: https://github.com/ESIPFed/sweet/issues
3. In the last, 30 minutes we'll take one request and demonstrate how we go about "ontologising" it in ENVO and how we link that to SWEET to create interoperable ontologies across the Earth and life sciences.

Come join us and help us shape the future of Geo-semantics!

Stuff you'll need:

1. A GitHub account available at https://github.com/
2. An ORCID (for nanocrediting your contributions) available at https://orcid.org

Notes for this session can be found at https://docs.google.com/document/d/1iupSeRRGmgjMBSjVAWIX0MFr1N4yr7DsIxnZhN3G1Zk/edit?usp=sharing

Session recording here.

Data repositories often rely upon conceptual models that provide formal representation information and identity conditions for digital resources -- for instance, the ontologies that underlie semantic data, or conceptual models like FRBR that underlie digital libraries. Though these later two cases represent extremely well documented conceptual models, there are many other instances where underlying conceptual models are tacit or inexplicit, and rarely published by practitioners and researchers. This makes it hard to build on one another's work, identify weaknesses in our models or modeling approaches, or forge new innovative collaborations. Furthermore, even in cases were conceptual models are well articulated, we believe there is a need for further discussion related to the methods used in modeling work, and the open research questions regarding conceptual modeling.

To that end, we'd like to see ESIP become a home for conversations about conceptual modeling for earth science data! We (https://sig-cm.github.io) are a group of information scientists who believe that sustaining a rich tradition of research and development in conceptual modeling in LIS requires collaboration with, and contributions from, communities like ESIP. This session would be the second in a series of interdisciplinary workshops, panels, and working sessions with the goal of building community and a research agenda around conceptual modeling work in libraries, archives, museums, and data repositories.

Plan for session:
- short lightning talks from presenters, setting the stage and outlining the topic
- a working session, in which participants are split into small groups to discuss areas of unmet need, and develop research questions, possible future research/development directions for ESIP + conceptual/data modeling efforts.

Session recording here.

The Disaster Lifecycle cluster is hosting a breakout session to explore sources of trusted datasets from various agencies and what constitutes operational readiness for these data. A key issue is how ‘Fitness for Use’ criteria can apply across ORLs, the Operational Readiness Levels.

With FEMA’s encouragement, collaborators at the All Hazards Consortium are “operationalizing” ORLs for data-driven decision-making support to improve situational awareness in response to power outages, transportation, fuel and lodging after major disasters. An interesting development is the need to assign fixed ORLs to datasets, rather than determining the ORL value based on specific use cases. The GIS ORL team within the Sensitive Information Sharing Environment (SISE) committee of the Fleet Response Working Group (FRWG) recognizes that latency, resolution, and coverage features have a significant impact on dataset readiness for most critical infrastructure and many weather and other EO datasets. However the inherent confusion that changing a trusted dataset’s ORL assessment creates a bigger problem for operator training and response efforts. Currently, most of their critical datasets are logistical in nature (what roads have been closed by state authorities, where can truck drivers get fuel/ food/ lodging, where are the authorized staging locations, etc.) and amenable to fixed ORLs assignments.

The recent wildfires in CA and associated mud and debris flows are impacting lives and property. Earthquake exercises are leading to data needs by decision makers that can drive situational awareness and decision making criteria. For example, soil condition information in burn scar areas is critical for NWS forecasters to know so they can accurately identify rainfall thresholds for issuing flood warnings in burn scar areas.

Looking forward to successfully using more trusted EO data for disaster operations, we plan to hear about current and planned datasets for disaster response needs. We are also seeking ways to clarify fitness for use criteria (especially latency, resolution and coverage) for these datasets that otherwise would meet the current readiness criteria of ORL1.

Agenda

Trusting Data Sets, Needs & Use Cases [60 min]

• [TRUST] Fleet Response WG approach to ORL - Chris McIntosh/ Bent Ear Solutions and All Hazards Consortium
• [NEED] NWS Hydrology needs for EO data - Katherine Rowden/NOAA NWS Western Region Hydrology
• [USE] Building a Community Based Housing Disaster Recovery GIS Application - Ashley Tseng/NCDP
• [RISK] Talk by Maximilian Dixon, Hazards and Outreach Program Supervisor at Washington State Emergency Management Division

Federal Agency Panel [20 min Conversation]
What are the issues and how to improve discovery and access to federal datasets across independent portals, ala Radiant.Earth?

• NASA Maggi Glasscoe (Remote) - Disasters GIS Team Lead 
• USGS Marie Peppler (Remote) - Emergency Management Coordinator, Acting
• NOAA Kari Sheets (Remote) - NOAA/NWS Geospatial Data Lead
• DHS / FEMA Chris Vaughan (Remote) - Geospatial Information Officer

Please note: Our panelists participated in our ESIP webinar on "Trusted Federal Data Sources for Hazard Response and Decision Making" which was recorded on June 24 to highlight current data services and upcoming plans. The Youtube video is available: https://youtu.be/ueUhJJIYCII

Session Takeaways

1. Clarifying local EOC approach to request federal resources (data and aircraft); more is needed to educate local and regional personnel.
   
2. Trust issues as handled by the ORLs - Determining operational readiness levels for federal data sets that powers 30-second decision making
   
3. Federal Data offerings for Disaster Response - Which agency has what information? There are at least 4 federal portals for emergency information. FEMA may be seeking a resolution, potentially an application that could tell what data has been collected and, if so, which agency has it?
   

Session recording here.