The NASA Earth Science Data System (ESDS) Program is responsible for the long term archival and distribution of NASA’s Earth Observing (EO) data, which is accomplished largely through the establishment and continued funding of 12 discipline-oriented Distributed Active Archive Centers (DAACs). The primary goal of this program-level directive is to maximize the scientific return from NASA’s EO data from missions and campaigns in support of research, applied science, decision makers, and society as a whole. To keep pace with increasing numbers of expected datasets provided by new EO missions and campaigns, largely driven by the recent Decadal Survey for Earth Science and Applications from Space (ESAS 2017), there has been an earnest push toward more sustainable architectures, interoperable frameworks, and on-demand user-tailored services to support this influx of both variety and volume of big EO NASA data. This presentation intends to focus on the motivations, current implementations, and planned deployment supporting big EO data architecture, frameworks and services from the perspective of the Physical Oceanography DAAC (PO.DAAC). Cloud storage architecture presents new opportunities to be exploited to the benefit of all data users, of which the following are being considered for development by PO.DAAC: cloud-based and on-demand data analysis/visualization tools and services, virtual machine environments to support legacy analysis software and toolkits, open-source and web-based development environments such as Python notebooks, open-source code sharing, APIs for more interactive data/metadata search and extraction, and an overall more consistent user experience in data tools/services with more predictable data download performance. Even though PO.DAAC’s cloud evolution is still in development, some headway has already been made toward frameworks and services that would readily extend levels of support for data within a cloud environment, including: 1) web services REST APIs for data granule subsetting and extraction, 2) web services toolkits and data recipes, 3) HTTPS-based WebDAV virtual drive mounting, 4) REST APIs for visualization and analytics, 5) advocacy for interoperable data/metadata formats (e.g., netCDF-4, HDF-5, JSON, and ASCII YAML) and metadata specifications (e.g., ACDD-1.3, CF-2.0, ISO-8601, ISO-19115), 6) leveraging internal compression capabilities for increased I/O performance within netCDF-4 and HDF-5 data files, 7) containerization of software via Docker, 8) consistent software version-control through GitHub, and 9) open-source licensing of user-oriented software and tools. Among many other key legacy NASA missions, the PO.DAAC is charged with supporting the upcoming Surface Water Ocean Topography (SWOT) mission. Pre-launch preparations for SWOT have placed the PO.DAAC on NASA’s leading edge for cloud storage and cloud computing architecture development and deployment, and represents a fundamental paradigm shift from legacy on-premise data frameworks and services. Much of this paradigm shift is driven by cost reduction due to the estimated day-to-day increase in volume ingress and egress that will increase by approximately an order of magnitude overnight just due to SWOT alone. The secondary motivator toward this new storage and distribution paradigm is the variety of data that will be provided by SWOT, forcing PO.DAAC to cater toward a much more discipline-diverse user base emanating at the outset of a single mission (e.g., physical oceanography, cryosphere, hydrology, etc.). Detailed survey analysis of targeted early adopters of SWOT data has revealed the potential for tremendous user uptake in cloud-based analysis and analysis-ready data tailoring services. Examples of current implementations will be showcased, along with an overview of technologies and frameworks that are planned for future deployment.

Big Data processing chains working on geospatial data such satellite imagery have been subject of research for many years and are now being operationalized in standardized ways. Well-defined interfaces and standardized data exchange formats have been developed that make working with satellite imagery end-user friendly. Recent advances in the standardization domain even support the ’applications to the data’ paradigm that allows executing arbitrary applications close to the physical location of the data. These approaches make use of standardized interfaces as defined by the Open Geospatial Consortium and combine them with container technologies such as Docker or Kubernetes. Both function on a variety of cloud platforms operated both by private industry and public organizations. Recent efforts in the standardization sector have even addressed an essential pillar towards commercialization of platforms, applications, and data that was missing in the past: quoting, billing, and security aspects.
Following the idea of a generic set of Earth Observation Exploitation Platforms that build a type of transparent and permeable platform ecosystem, a number of Thematic Exploitation Platforms (TEPs) as well as Mission Exploitation Platforms (MEPs) have been made available on cloud platforms over the last couple of years. ESA got engaged with the Open Geospatial Consortium as a standardization organizations to ensure that both types of platforms provide standardized interfaces that facilitate efficient and seamless communication between a variety of clients and the platforms. These efforts have produced two Web Processing Service (WPS) profiles named Execution Management Service (EMS) and Application Deployment and Execution Service (ADES). The Execution Management Service (EMS) represents the front-end to both application developers and consumers. It makes available an OGC Web Processing Service interface that implements the new resource oriented paradigm, i.e. provides a Web API that follows REST principles (WPS v3.0) and uses JSON as serialization format. This API supports the registration of new applications. The applications themselves are made available by reference in the form of containerized Docker images that are uploaded to Docker Hubs. These hubs may be operated centrally by Docker itself, by the cloud providers, or as private instances that only serve a very limited set of applications. Initially developed to deploy applications only, the EMS is now emerging into a workflow environment that allows application developers to re-use existing applications and orchestrate them into sequential work-flows that can be made available as new applications again. This process is transparent to the application consumer.
The execution platform, which offers EMS functionality to application developers and consumers, acts itself as a client to the Application Deployment and Execution Services (ADES) offered by the data storing cloud platforms. The cloud platforms support the ad-hoc deployment and execution of Docker images that are pulled from the Docker hubs using the references made available in the deployment request. Once application consumers request the execution of an app, the exploitation platform forwards the execution request to the processing clouds and makes final results available at standardized interfaces again, e.g. at Web Feature Service (WFS) or Web Coverage Service (WCS) instances. In the case of workflows that execute a number of applications sequentially, the exploitation platform realizes the transport of data from one process to the other. Upon completion, the application consumer is provided a data access service endpoint to retrieve the final results. All communication is established in a web-friendly way implementing the emerging next generation of OGC services known as WPS, WFS, and WCS 3.0.
Lately, quoting and billing information has been added to the Execution Management Service (EMS) and the Application Deployment and Execution Service (ADES). The new feature allows application consumers to get an indication of costs caused by specific application requests. The billing feature is an important step towards commercialization of data, applications, and platforms. This presentation provides an overview of the latest standardization efforts that will facilitate earth observation Big data cloud processing and commercialization.

In the past, infrastructure limitations mandated that if one wants to do efficient analysis on EO data, one needs to pre-process it to the common structure – create so called “DataCube”. This pre-processing usually involves re-gridding, re-projection and some more advanced steps such as atmospheric correction, orthorectification, etc. This step is costly, both from processing as well as storage perspective as it commonly required replication of data. One can leverage parallel processing in a way to eliminate additional storage, but overall costs remain. These limitations commonly result in reduced data coverage, either spatial (e.g. country instead of world) or temporal (only one year). Another disadvantage of this approach is inflexibility – whenever an improvement in the process is implemented, one has to re-process complete archive.

Nowadays, with cloud infrastructure rampant and cloud-optimized data formats appearing, the term “DataCube” is losing relevance. Or rather, it is shifting one layer down – to individual scenes. Cloud-optimized GeoTiff (COG) has all necessary elements of the data cube – internal tiling, overview layers, location data in the header, etc. Sentinel-2 native JP2000 contains a pyramid (based on wavelet transform) as well, and NetCDF/HDF format also. Match this fact with recent "standards" to process data to L2A on ground segment level (e.g. ESA) and having data available on the cloud (DIAS, AWS, GCP, etc.) and what one data analysts needs is a "data driver" rather than "DataCube". Due to different formats of the data, meta-data and other specifics, it is inconvenient for a random user to access it in an efficient manner. The task of the "DataCube operator" is therefore to adapt their data access engine to various data sources rather than to pre-process these data sources to match their processing engine.

A technical solution demonstrating above described approach is Sentinel Hub, which is around for a bit longer than two years now, more or less as long as Sentinel-2 is operational. It is used by hundreds of applications and tens of thousands of end-users, processing more than two million requests per day. Average response time is 0.7 seconds, which is more than fast enough for vast majority of users. In this time Sentinel Hub executes all the typical DataCube steps - find relevant scenes addressing user's request, read necessary data, re-project and re-grid to specified parameters, create a composite or statistical analysis, etc. Data access interfaces are compliant with commonly used OGC standards (WMS, WCS, WMTS, etc.) so that it is easy for users to integrate these in their existing environments.

We will present how Sentinel Hub is handling federation approach, running on various platforms in parallel (AWS, Mundi, CreoDIAS, ONDA), long time-series analysis, how annual cloudless mosaic can be created ad-hoc and how it is possible to engage data fusion to combine various data sources, e.g. performing on-the-fly orthorectification of Sentinel-1 GRD data.

Special focus will be put on lessons learned, providing guidance on good and not as good examples of existing data structure, most commonly used functionality by end-users, limitations of existing standards and how to work around them as well as some thoughts on possible business models and how one benefits best with open-source approach.

There are now several Earth observation satellites in orbit around our fair planet, capturing and beaming down huge volumes of broad-spectrum multi-band imagery every day – imagery that is both aesthetically beautiful and of deep scientific interest to a broad range of researchers. High resolution multi-band imagery for the whole world has been captured on regular basis since the early 1980s (Landsats 4 and 5) providing a rich source of data for the study of seasonal and longer-term changes in climate, geology, hydrology and land use.

Digital Earth Australia is a platform managed by Geoscience Australia to make Earth observation satellite imagery of Australia readily available to Australian government, industry and research users. At the heart of Digital Earth Australia is the Open Data Cube, an internationally developed open source Python platform designed to manage Earth Observation satellite data and to support interactive data science and scientific computing.

CSIRO’s Data61 has been working with Geoscience Australia to develop open source web services and clients to make this rich source of data more accessible to both expert users and the general public. A light-weight Python web application server built on top of the Open Data Cube and optimised for cloud deployment is under active development and is capable of publishing Earth observation satellite data using standardised geospatial web service protocols such as Web Map Service (WMS) and Web Coverage Service (WCS). When combined with the TerriaJS web client (a Data61 developed open source framework for exploring and visualising web-based geospatial data), this results in a complete software suite for managing, publishing, exploring and viewing Earth observation data and imagery, all available for use and collaborative development under free and open-source licences.

This presentation will discuss the progress of the project to date and our plans for the future. Early wins and unexpected obstacles will be identified. The advantages and shortcomings of the OGC WMS and WCS standards will be discussed from the points of view of both the implementer and the consumer. The strengths and limitations of using the Open Data Cube in a cloud-based deployment environment will also be addressed.

In 2015, all United Nations (UN) Member States adopted the 2030 Agenda for Sustainable Development. The Agenda provides a shared blueprint for peace and prosperity for people and for the planet, considering our current situation and helping to create a plan. The core of this agenda is a set of seventeen Sustainable Development Goals (SDGs), which represent an urgent call for action by all countries - both developed and developing - in a global partnership. Collectively, all the countries have recognized that ending poverty and other deprivations must be done in parallel with strategies that improve health and education, reduce inequality, and trigger economic growth – all while handling climate change and working to preserve the planet's oceans and forests.
The Open Data Cube (ODC) initiative aims to increase the value and impact of global Earth Observation (EO) satellite data by providing an open and freely accessible data exploitation architecture and to foster a community to develop, sustain, and grow the technology and expand the breadth and depth of its applications for societal benefit. Successful ODC implementations exist in 9 countries and are under development or consideration in another 50 countries. Since its inception in early 2017, the ODC has made significant progress in the advancement of open source software tools and algorithms that support the deployment and operation of data cubes on local or cloud computing systems around the world.
The Committee on Earth Observation Satellites (CEOS) System Engineering Office (SEO) has supported the ODC initiative to provide a data architecture solution that has value to its global users and increases the impact of EO satellite data. The ODC is an open-source platform for processing satellite data. The software development team at CEOS SEO has developed a set of algorithms, software products and tools around the core ODC. The CEOS SEO team has developed and released a large collection of Jupyter notebooks that act as interactive Python development environments - allowing developers to divide their code into blocks which can be run independently of each other, with variables stored in the background and the environment persisted between blocks. The notebooks have been instrumental in providing hands-on training to many international users in the remote sensing community and have been presented at multiple conferences, tutorials, training sessions, and international presentations.
The CEOS SEO team has recently developed and released a set of innovative notebooks addressing UN SDGs 6.6.1 (spatial extents of water-related ecosystems), 11.3.1 (ratio of land consumption rate to population growth rate), and 15.3.1 (proportion of land that is degraded over total land area). These notebooks empower users by providing features that will assist with streamlining analysis ready data retrieval, processing, and visualization. The main contributions in this paper are: (1) briefly describing the framework of the UN SDG notebooks, (2) enumerating the notebooks’ salient features, and (3) discussing current limitations and proposing approaches to overcome these limitations.
The UN SDG 6.6.1 notebook focuses on SDG Indicator 6.6.1.i – spatial extents of water-related ecosystems. It is divided into two sections, the first of which analyzes a single time period over a given region. It examines the mean of water presence over time, minimum and maximum water extents, and frequency of water classification for each pixel. The second section analyzes two time periods - a baseline and an analysis time period. We determine the change in water from the baseline period to the analysis period and finally classify the area as one of 5 classes defined by UN SDG 6.6.1. For the time series plot of the mean of water, we plan to bin our results monthly or yearly. We will consider pixels to be water if it was water at any time during the time period. We have discovered that sparse temporal coverage can make side-by-side comparison of minimum and maximum water extent difficult to interpret for some regions, so we plan to use monthly or yearly mosaics of the data.
The UN SDG 15.3.1 notebook analyzes the proportion of land that is degraded over total land area. Land classes are defined by the ESA CCI (European Space Agency Climate Change Initiative), which provides yearly data from 1992 to 2015 at a resolution of 300x300 meters. The ESA CCI land classes are converted to 6 IPCC land classes, and a random forest classifier is trained on as much data from 2000 to 2015 as possible. The classifier learns to predict land classes based on measurements in the input data, ignoring derived data such as quality bands. The predicted and actual land classes are compared for the first and last years, a change matrix is created which shows the change in land classes between the first and last years, and the Normalized Difference Vegetation Index (NDVI) is shown by class and over time along with a Mann-Kendall regression. We plan to add an image of the number of land changes for each pixel between the first and last year as a heatmap of change frequency. We also plan to label “degraded” pixels (those that ever experienced a land cover change labeled as degradation). In future, we will allow users to specify which land cover changes are considered degrading, neutral, or positive changes, and color the resultant change matrix accordingly.
The UN SDG 11.3.1 notebook is implemented to determine if land is being sustainably consumed. The sign of unsustainable land consumption, per SDG 11.3.1, is urban-land size growth happening at faster rate than population growth. This notebook determines both population growth and land consumption growth for a specified area. These two are then used to derive a simple ratio that can provide valuable insight into population growth. The notebook currently accepts a shapefile that details the "shape" or geographic boundaries of a study area. Included in this file are population estimates for two years. For those years, a 'built-up' index with a threshold is used as a proxy for urbanization. While not an exact measure for urbanization, built-up does correlate with the existence of populated land. Currently, users need to provide their own population data, which can be tedious and has the potential to misalign with the land change data. We are currently working towards ingesting the Gridded Population of the World (GPW) version-4 data, allowing a user to query a global inventory of ingested population data directly from the ODC. This revision will also make the notebook easier to validate on different regions.
The UN SDGs are a universal call to action to end poverty, protect the planet, and ensure that all people enjoy peace and prosperity, established by the UN as part of their 2030 agenda for Sustainable Development. With the increased interest in data cubes, the ODC is poised to become a world leader in open source data cube technologies to enhance the value of satellite data for all users by providing insights into many of the UN SDGs.

With the opening of global archives of Earth Observation data streams from satellites we have arrived at a richness of operationally available observations over the whole globe, starting from the Landsat series of satellites and now the plethora of available data coming through Copernicus and its series of Sentinel satellites, that has never been available before. This created huge opportunities for research and businesses, being able to exploit the temporal domain of those observations in a powerful manner, but also poses challenges in terms of data management and processing capacities. As a consequence, a growing number of cloud services and customized solutions in various research centers have been developed, leading to processing workflows optimized for specific system architectures and back-end infrastructures. As such this is limiting portability and reproducibility of workflows across back-ends, both for science and business applications.

OpenEO, a Horizon 2020 project (grant 776242), addresses this problem by defining an API between service provider back-ends and client applications. This API can be seen as a common language that defines interfaces for finding, accessing, processing and retrieving data, and only requires that a back-end architecture specific driver and corresponding client libraries are used when defining the workflows. The project however does not stop at simply defining an API, but also provides implementations for a number of different back-end solutions, ranging from a file-based storage system with processes running in individual containers, over data cubes exposed via OGC web services to GRASS GIS, the Google Earth Engine and the Sentinel-hub. On the client side, libraries are developed in python, R and javascript, leaving space to develop applications based on Earth Observations with very different requirements, from web visualization to extensive time series analysis in a research setting. In addition to a large set of simple processes, user-defined functions allow users to submit more complicated processes as python or R code, e.g. for dedicated time series models, to be run on the imagery data.

All drivers both on client and back-end site are developed as open source software and can be found on the project’s Github group ( https://github.com/Open-EO ). This enables everybody to peek into the available implementations and develop own adaptations based on what is needed for specific back-end solutions. There is also no limitation to define new client libraries in other languages or develop plugins for common software such as QGIS or the Sentinel Toolboxes.

A proof of concept showing the successful interaction and processing of different clients and back-ends has been published ( http://openeo.org/openeo/news/2018/03/17/poc.html ).

During LPS, details about the concept and the current status of the development will be presented, together with examples from use cases implemented with openEO.