The key to sustainable development is achieving a balance between the exploitation of natural resources for socioeconomic development, and maintaining ecosystem services that are critical to human’s wellbeing and livelihoods. Some of these environmental issues can be monitored using remotely-sensed Earth Observations (EO) data that are increasingly available from freely and openly accessible repositories. However, the full information potential of EO data has not been yet realized. They remain still underutilized mainly because of their complexity, increasing volume, and the lack of efficient processing capabilities.
Following the work done in Australia, the Swiss Data Cube (SDC – http://www.swissdatacube.ch) is an initiative supported by the Federal Office for the Environment (FOEN) and developed, implemented and operated by the United Environment Program (UNEP)/GRID-Geneva in partnership with the University of Geneva (UNIGE), the University of Zurich (UZH) and the Swiss Federal Institute for Forest, Snow and Landscape Research (WSL). The objective of the SDC is to support the Swiss government in environmental monitoring and reporting, as well as enable Swiss scientific institutions to fully benefit from EO data for research and innovation. Currently, the SDC contains 35 years of Analysis Ready Data of both optical (e.g., Landsat 5, 7, 8; Sentinel-2) and radar (e.g., Sentinel-1) satellite data over the entire country (total volume: 5TB; 200 billion observations). The SDC is an innovative analytical framework, based on the Open Data Cube software stack, lowering the barrier caused by Big Data challenges (e.g., Volume, Velocity, Variety) and provides access to large spatiotemporal data in an analysis ready format. It allows users to benefit from this new generation of EO data, and in particular minimizing the time and scientific knowledge required to access, prepare and make it possible to analyze large volume of data having consistent and spatially aligned, calibrated observations.
The SDC is aiming at contributing to the national “Digital Switzerland” strategy by (1) supporting innovation and growth; (2) improving efficiency and effectiveness of government investments; (3) improving management of natural resources; (4) stimulating research; (5) generating decision-ready information products; and (6) improving data access and use & enabling new products/services that can transform everyday life.
Ultimately, the SDC will deliver a unique capability to track and monitor enviromental changes in unprecedented detail using Earth observation satellite data and enable more effective responses to problems of national significance. It can provide the long baseline required to determine trends, define present, and inform future.

The datacube model has rapidly gained acceptance as a cornerstone for analysis-ready data, and also the corresponding service model which is more powerful, but easier to use than existing API-based interfaces. Mature and widely adopted datacube standards show the way how datacube functionality can be presented to clients, be they human or m2m connections. Specifically, the OGC Coverage Implementation Schema (CIS) data model and the Web Coverage Service (WCS) service model suite define a frame-work adopted by the main open-source as well as proprietary tools, including MapServer, GeoServer, GDAL, QGIS, ArcGIS, and python/OWSlib.

While the basic, most widely used functionality includes access, possibly extraction, and reformatting of a data sub(set) for download, processing and in particular data fusion represent complex, resource-intensive challenges. In the WCS suite this is addressed through a concept of modularity where WCS Core (which is mandatory for any WCS to implement) offers the GetCoverage request to accomplish “give me part of this spatio-temporal coverage, in my favourite format” and (optionally implementable) extensions add further functionality facets, up to the spatio-temporal datacube analytics language, Web Coverage Processing Service (WCPS).

For example, computing the NVDI over Europe, on the 1st of July 2018, using Sentinel data stored in a datacube with the same name, and returning the result as a NetCDF file, can be achieved with the following WCPS query:

for $c in (Sentinel)
return encode( (($c.nir – $c.red) / ($c.nir + $c.red)) [ Lat(35:70), Long(10:40), time(“2018-07-01”) ], “image/tiff” )

As WCPS allows combination of datasets (“joins” in database terminology) the question arises: what if datacubes to be combined reside on different computers (cloud scenario), or even in different data centers (federation scenario)? A suboptimal implementation obviously could cause massively degraded performance.

In this contribution we present federation methods implemented in the rasdaman datacube engine which is accepted technology leader, standards driver, and reference implementation for datacubes.

RASDAMAN ARCHITECTURE

The rasdaman engine resembles a fully-fledged Array Database System which utilizes a declarative query language enhancing standard SQL with high-level array operators; in fact, ISO in 2018 has adopted this language as an extension to the SQL language standard. This datacube language is domain agnostic and can serve as well medical imagery, cosmologic simulation results, and the like. A semantic layer on top of it offers geo semantics, i.e., it knows about spatial and temporal coordinates and, hence, also about regular and irregular grids. This semantics is offered via OGC WCS, WCPS, and WMS interfaces so that a plethora of clients can utilize massive datacubes managed by rasdaman. All such requests are translated to array SQL queries internally.

In the server, such incoming queries are optimized, parallelized, and ultimately mapped to datacubes that are tiled on disk (or tape). As opposed to standard regular tiling (i.e., equi-sized tiles) rasdaman allows for arbitrary tiling guided by several strategies. Tiling remains invisible to the applications (and, hence, in the queries), it is a tuning parameter for the administrator. Tiles of the same datacube can even sit on different machines.
The rasdaman server is multi-parallel per se, allowing any number of simultaneous parallel requests (inter-query parallelization) as well as parallel execution of incoming queries (intra-query parallelization).

Datacube operations are usually triggered via standardized web services, such as WCS, WMS and WCPS. However, these services are only the interface to the client performing the operation, while the processing itself is handled at a lower level which enables optimized, efficient execution. In rasdaman, the geo layer intercepting the web requests representing datacube operations translates them into array database queries, which are then handled by rasdaman’s query processing engine.

The engine analyses each incoming query and compiles it into machine code. During that process, a series of optimization steps is applied, one of which is distribution.

Such federations are being done routinely meantime. In the EarthServer project, Petabyte-scale datacubes have been federated across ECMWF/UK and NCI/Australia. Another real-life use-case of the optimization is given in the federation which was set up between the German Sentinel hub, CODE-DE, established in the BigDataCube project, and the Alfred Wegener Institute for Polar and Marine Research (AWI). Among others, temperature datacubes are available at CODE-DE, and SeaIce datacubes are available at AWI. In one of the use-cases, scientists need to compute the sea ice distribution at different temperature intervals. The WCPS query that achieves that is presented below. The datacubes involved contain data at different resolutions and in different coordinate systems, which gets adjusted in the course of query processing. Most recently, FCU/Taiwain has been federated with CODE-DE.

In summary, rasdaman allows federated processing of declarative queries whereby complete location transparency is given: any federation member can receive the query, and the federation will dynamically orchestrate each query individually for optimized processing.

ACKNOWLEDGMENT

This work is being supported by H2020 LandSupport, H2020 EOSC-hub, and German BMWi BigDataCube.

The Earth System Data Lab (ESDL) provides researchers with a comprehensive and analysis-ready data set, which comprises relevant variables for Earth System Sciences and a Virtual Laboratory to readily build multi-variate workflows. The project has adopted a truly user-centric approach striving for the implementation of many different use cases to maximise the scientific outcome of this activity. To this end, four Champion Use cases involving recognized scientific institutions have been identified and implemented using ESDL. In addition, more than 20 Early Adopters have been using the ESDL to conduct self-consistent scientific research projects. In this presentation, we give an overview of the diversity of successfully implemented use cases and show highlights of their results. Moreover, we synthesize our experiences with operating a Virtual Lab with the Earth System Data Cube and summarize the lessons learnt over the past years as a service provider.
Earth System Science is concerned with the interactions of Earth’s different subsystems by adopting a holistic approach to understand the complex dynamics of our environment. It is inherently global and multi-disciplinary, involving biology, chemistry, physics, mathematics, and applied sciences and thus a very data hungry science. Typically, holistic and simultaneous observations of several variables on a global scale are needed to study and understand the complex interactions witnessed on our planet. Spatial coverage, sampling frequency and consistency makes Earth Observation indispensable for the Earth Sciences as its observations nowadays allow for deriving a wealth of relevant parameters that characterize the atmosphere, hydrosphere, cryosphere, and biosphere. Consequently, Earth System Scientists often spend a lot of time for acquiring the observational data sets required for their research. “Acquisition” here stands for several tasks, e.g. finding adequate data sets, getting access to them, downloading, storage, quality control, resampling, updating and others. All these steps are costly and become easily unmanageable for a researcher if multiplied by a number of variables – and have nothing to do with advancing the insights into the Earth System. They are hence dispensable and distract the researcher from dedicating time to his or her core activity.
The ESDL project is ESA’s answers to these challenges. It is a low-threshold offer to greatly increase a researcher’s efficiency and fosters innovative science, because trying out scientific ideas becomes much cheaper if researchers do not have to invest in preparing data and computational infrastructure before actually starting their scientific endeavour.
To this end, a data cube of more than 80 variables with global coverage and a temporal coverage of at least 10 years per variable is maintained and made available in the cloud to registered ESDL users. Most of these data sets have been derived from Earth Observation, e.g. from ESA’s CCI projects, but some are generated by models, e.g. ECMWF’s ERA-5. All data sets are transformed to a common grid and the xarray data model, which is effectively evolving into a quasi-standard for multi-dimensional data analysis with growing popularity across different Earth Science disciplines, and hosted in the cloud as data cubes in different temporal and spatial resolutions.
While the storage in object-storage also allows for remote access at no costs for users, the preferred way to interact with the Earth System Data Cubes (ESDC) is via the Virtual Laboratory implemented as a JupyterHub service, which provides users with a read-eval-print- loop (REPL) environment in the browser for the programming languages Python and Julia. The JupytherHub automatically instantiates a new JupyterLab session after login and terminates them after log-out or shut down of the browser window. While Python is arguably the most commonly used programming language among natural scientists, which provides a wealth of packages for data analysis and processing, the relatively new programming language Julia rapidly gains popularity among data scientists for its efficiency and speed in big data applications.

Significant, global, environmental change is predicted by the end of the century whereby warming will likely exceed the 2⁰C threshold specified within the Paris climate agreement. Climatic changes act upon the individual with organisms forced to migrate, adapt in situ to changing conditions, or to face (local) extinction. As a response to shifting environmental gradients, ecosystem functioning and associated ecosystem services are therefore threatened. Understanding how particular ecosystems respond to changing conditions is a pressing concern within ecology, required to accurately predict future changes and thus to devise appropriate adaptation or mitigation strategies. A notable problem, however, is encountered in the lack of empirical evidence – recent global climatic change is not comparable to that expected by 2100 and observational studies consequently fail to capture the potential consequences of a warmer world.

Whilst average temperature changes on a global scale may remain limited thus far, patterns of warming are regionally much more variable. For example, parts of the Arctic have already experienced rapid warming of ~5⁰C resulting in large impacts upon individual plant traits and the structure of plant communities. ‘Arctic Greening’ has been confirmed in both ground and satellite-based studies, with encroachment of shrubs and increases in productivity as a response to increasing temperatures. This phenomenon has been shown to be highly heterogeneous within the Arctic with divergent responses by both species and region, and the temperature-greenness relationship complicated by other abiotic factors such as snow cover and local topography. An understanding as to the nature of abiotic-biotic-climatic interactions is therefore essential to accurately predict the trajectory of future ecosystem change under different climate change scenarios.

Geodiversity – the diversity of abiotic nature including geomorphology, pedology, geology and hydrology – has increasingly been recognised as an important factor within conservation. The ‘Conserving Nature’s Stage’ approach posits that by protecting the diversity of abiotic nature (geodiversity), biodiversity can be preserved too, even if specific ‘actors’ (species) change. Indeed, geodiversity has been shown to support ecosystem processes and biodiversity of alien and native species across a range of habitats, driving differentiation at the landscape scale by increasing available niche space (e.g. through the provisioning of microclimates). Despite the growing consensus surrounding its importance, empirical evidence does not yet exist as to the role of geodiversity in buffering against recent climate change. Where geodiversity is high, the presence of microclimates (for example) may allow favourable conditions to persist even in areas of rapid warming, buffering individuals against the effect of climate change. Through the explicit consideration of the different geofeatures which comprise geodiversity meanwhile (e.g. landforms, or soil types), the responses of individual species to climate may also be better characterised.

This forms the basis for this study – examining whether Geodiversity buffers against changes in the Gross Primary Productivity (GPP) of plant communities within the Arctic. Through the use of datasets hosted upon the Earth System Data Lab’s (ESDL) Earth System Data Cube (ESDC), changes in Arctic (latitudes of >60⁰N) GPP and climate (temperature/precipitation) were examined for the period 2000-2015. These were then integrated with different geofeatures (and geodiversity metrics) using Boosted Regression Trees (BRTs) to answer three main questions:

1. Does the inclusion of geofeatures into models improve our understanding of ‘Arctic Greening’?

2. Are areas of higher geodiversity less vulnerable to the effects of climate change?

3. Does this relationship hold true across different spatial scales?

A secondary aim meanwhile was to assess the effectiveness of the ESDL data cube for this study, whether the pre-aligned (in space/time) datasets were sufficient, and what additions the system could benefit from.

In total, five key facets of geodiversity were considered: pedology, geology, hydrology, topography and geomorphology across 4 different spatial scales (0.083⁰, 0.125⁰, 0.25⁰ and 0.5⁰) and two different ecoregions (boreal forest/tundra), with the relationships between individual variables and GPP visualised via partial dependency plots. Initial results indicate that the inclusion of specific geofeatures improved the performance of BRTs in characterising changes in GPP (compared to including just climatic variables), though the strength and nature of relationships with different geofeatures varied by ecoregion. The ESDL meanwhile provided useful datasets that dramatically reduced the time required for preprocessing data, though the gathering of additional datasets was still required to derive geofeatures. Overall, these results suggest that better characterisation of the abiotic environment (incorporating a wide range of geofeatures) is required to understand the responses of plant communities to climate change, where these results could be applied worldwide to predict changes in other areas that have not yet experienced as rapid climatic change as the Arctic.

To increase the uptake of synthetic aperture radar (SAR) data it is crucial to ease the preprocessing thereof.
Optimally, a user is presented with a standardized data cube ready for analysis without further knowledge of SAR processing details.
pyroSAR addresses this need by providing a complete workflow from retrieving the raw data from its provider to filled data cubes ready for analysis.
pyroSAR is harmonizing the metadata of multiple SAR data formats for relevant parameters like acquisition time or relative orbit.
It keeps record of registered scenes with their metadata and of the already preprocessed data.
This eases the management of SAR data for multiple testsites in multiple projects.
Currently pyroSAR is using the ESA Sentinel Application Platform (SNAP) or the Gamma SAR software for the preprocessing of the SAR data.
The multiple parameters of SAR processing and the different approaches are hereby broken down in simple python functions which select the needed processing steps based on the user defined requirements like pixel size, study area and variable type .
Currently the preprocessing functions are optimized for the conversion of Sentinel-1 IW GRD products to geocoded Gamma0.
This includes the subsetting on a given test site geometry, border noise removal, Range-Doppler geocoding and topographic normalization. This data can then be scaled to dB.
The preprocessed data sets will then be provided in a data cube from the open data cube (ODC) initiative or from the Earth System Data Laboratory (ESDL) .
pyroSAR acts as a central SAR data broker to keep track of what data is available in raw or processed format and organizing this data in an easily accessible form while reducing the need to learn different processing software solutions thus leaving more time for actual data analysis.
To achieve this, pyroSAR goes beyond processing SAR scenes and provides a comprehensive toolchain for general spatial data handling connecting several open source software tools along the way.
It creates a framework around existing SAR processing tools to ensure consistent output from each of them reducing their individual handling to a minimum.