The European Space Agency is supporting the paradigm shift to move the software to the data with its SentiNel Application Platform, SNAP. SNAP is known as a toolbox for visualisation, analysis and processing of raster EO data, specifically for Sentinel and Envisat data. SNAP comes with 3 user interfaces: SNAP Desktop for interactive work with EO data, the Command Line Interface (CLI), which allows batch processing, and the Application Programming Interface (API) which allows to call SNAP from own software programs. The Graph Processing Framework (GPF) of SNAP is a key technology running in the background of all three interfaces, which allows to connect sequences of operators, lazy execution, in-memory processing sequences and automated parallelisation. In particular the GPF makes SNAP ideal for efficient operational processing of large datasets, or NRT processing, in a cluster or cloud system.

SNAP is deployed on many EO cloud systems: it is available on all ESA Thematic Exploitation Platforms and all Copernicus DIASes. It is also available on the German National Collaborative Ground Segment (CODE-DE) and potentially other EO cloud systems not known to the authors.

Interactive data users can launch SNAP Desktop in a virtual machine on the cloud and have immediate access to all data provided by the respective system. However, the real strength of SNAP in a cloud or cluster environment is revealed when automated processing graphs are set-up and integrated into a processing management system (which is provided individually by the respective cloud system). Together with the access to the data archives in the cloud or on the cluster, large processing jobs can be executed or a systematic, data driven production can be implemented and executed. A typical development cycle tests the processing graph interactively in SNAP Desktop and executes the (generalised) graph in CLI mode on large datasets or NRT mode.

The Apache Hadoop based Calvalus system for massive parallel processing of EO data is designed to optimally support SNAP’s Graph Processing Framework and thus is most suitable as processing system on a cloud or cluster infrastructure. Calvalus with SNAP is available to users on several public and private clouds and clusters, such as three national Copernicus collaborative ground segments (Germany, Finland, Estonia), the ESA Urban TEP and institutational clusters in Germany, Ireland, UK and Italy. Brockmann Consult is operating Calvalus on a private cloud with 500 processing nodes and 2PB online storage. SNAP has been used here for very large processing jobs generating ESA Climate Change data for Land Cover, Fire and Water Vapour, and is running operationally NRT to provide services to Brockmann Consults customers. Special Calval functions with SNAP on Calvalus allow fast match-up extraction, time series analysis, as well as prototyping and testing of algorithm changes.

In this presentation we present the capabilities of SNAP on EO cloud platforms at examples of the Urban TEP and the German CODE-DE platform. At the example of the Calvalus private cloud at Brockmann Consult we discuss advantages as well as shortcomings of SNAP on clusters in several operation modes, and conclude with recommendations for future work to better support EO data users with instrument toolboxes in a cloud and/or cluster environment.

Since 2014, the Copernicus Sentinel missions funded by the European Commission (EC) and operated by the European Space Agency (ESA) have been providing an unprecedented amount of diverse and high-quality data for Earth Observation (EO). The data volume rose from about 200 TB in 2014 (one operational mission) to above 7 PB in 2018 (seven operational missions) and will increase further with the planned launch of new Sentinel satellites. Thank to EC’s free and open data policy, legal and financial barriers in accessing the data have been removed. However, in practice the development of applications based on multi-temporal, multi-sensor datasets over large geographical regions remains very challenging. Next to expert knowledge about the individual sensor characteristics and modeling processes, a significant amount of IT infrastructure is essential in order to unlock the potential of EO Big Data for user applications.

The response to this challenge has been to bring software solutions and computing capabilities where the data are stored. Different implementations of this approach have been adopted by a number of private (e.g. Google Earth Engine, Earth on AWS) and public cloud platforms (e.g. ESA’s Data and Information Access Service (DIAS) projects), including the collaborative approach developed by the Earth Observation Data Centre for Water Resources Monitoring (EODC, eodc.eu). EODC is a public-private partnership with an initial stage connection to the WEkEO DIAS activity that is jointly established by EUMETSAT, ECMWF and Mercator Ocean. EODC hosts a Petabytes-sized global archive of selected level 1 products from the Sentinel 1, Sentinel 2 and Sentinel 3 missions. Additionally, it offers tools to generate higher-level products (e.g. Sentinel-2 Value Adder, SEN2-AGRI) on a cloud platform or HPC environment, such as the Vienna Scientific Cluster 3 (VSC3) and its upcoming upgrade VSC4. Finally, EODC provides a number of high-level software solutions that help reducing the entry barrier to EO Big Data, simplifying data discovery, access and processing at scale.

In this contribution, we focus on software tools that, connected to EODC’s data archive and computing environments, empower users to exploiting EO data in a friendly manner. These solutions include, e.g., a queryable data catalogue, a QGIS pugin and a JupyterHub server allowing to quickly visualize data or create personalized notebooks/scripts. Furthermore, we present a software package that provides a unified interface to access/process all types of Sentinel data in the archive, including with user-defined algorithms. This software is available also via the open-access interface developed by the openEO consortium (openeo.org). All solutions are showcased with practical examples of processing chains based on different sensors (e.g. radar and optical), implemented in EODC’s modular and scalable cloud infrastructure.

Google Earth Engine (GEE) is the computing platform recently released by Google “for petabyte-scale scientific analysis and visualization of geospatial datasets”. Using a dedicated High-Performance Computing (HPC) infrastructure, it enables researchers to easily and quickly access more than thirty years of free and public data archives, including historical imagery and scientific datasets, for global and large-scale remote sensing applications. In this way, many of the limitations related to data downloading, storage and processing, usually occurring when such large amount of information (Big Data) is analyzed, are effortlessly overcome [1] [2].

The leading idea behind this work is indeed to leverage the GEE platform capabilities in order to evaluate precision and accuracy at large scale of three free global Digital Surface Models (DSMs), ASTER GDEM, SRTM and the new AW3D30 global DEM. These GDEMs are already available inside the GEE catalog, ready to be easily imported and processed into the shareable scripting environment of the platform [2].

Therefore, accordingly to previous studies [2], [3], proper scripts were implemented inside the GEE Code Editor to compute standard statistical parameters able to represent DSM precision and accuracy (i.e. mean, median, standard deviation, NMAD, LE68, LE90). Specifically, the geometric accuracy assessment of the three global DSMs was performed on the wide territories of four American States (Colorado, Michigan, Nevada, Utah) and one Italian Region (Trentino Alto- Adige, Northern Italy), characterized by different terrain morphology, land covers and slopes. The assessment was carried out using two different reference DSMs: the USGS National Elevation Dataset (NED) for the American states and a LiDAR DSM for the Northern Italian Region. The standard statistic parameters were computed both at global scale (considering the whole State/Region) and in function of the terrain morphology using several slope classes (see Figure 1).

Overall, all the results achieved are quite consistent, showing a good accordance in their behavior. In particular, the accuracy of each considered DSM decreases with the increase of the slopes but according to different rates in function of its generation method.In conclusion, the results highlight the GEE potentialities to perform a large-scale DSM assessment in an efficient and easy way and, at the same time, the possibility to leverage such platform to generate an overall DSM obtained merging the different elevation data in function of their accuracy, and thus in function of the slopes of the area analyzed.


REFERENCES
[1] N. Gorelick, M. Hancher, M. Dixon, S. Ilyushchenko, D. Thau, and R. Moore (2017). Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment. 2017, 202, 18–27, doi: 10.1016/j.rse.2017.06.031.

[2] A. Nascetti, M. Di Rita, R. Ravanelli, M. Amicuzi, S. Esposito, and M. Crespi (2017). FREE GLOBAL DSM ASSESSMENT ON LARGE SCALE AREAS EXPLOITING THE POTENTIALITIES OF THE INNOVATIVE GOOGLE EARTH ENGINE PLATFORM. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, doi: doi:10.5194/isprs-archives-XLII-1-W1-627-2017.

[3] K. Jacobsen. Characteristics and accuracy of large area covering height models (2013). The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 40(1W1), doi: 10.5194/isprsarchives-XL-1-W1-157-2013.

BETTER is a 3-year H2020 project started in November 2017 that is implementing a Big Data intermediate service layer based on the development of data processors that deliver continuous data streams customized to user requirements. These customized solutions, denominated as Data Pipelines, are driven by 36 Data Challenges defined by users from key Societal Challenges sectors. The Data Pipelines will facilitate the downstream usage of large volume and heterogeneous datasets, so that users can focus on higher value tasks, such as the extraction and analysis of the potential knowledge within the data, and not on the processing of the data itself. Their main purpose is the continuous delivery of large volumes of higher level EO products to users, customized to their needs, and with that bring more downstream users to the EO market and maximize exploitation of Copernicus data and information services.

This paper presents the BETTER project concept and objectives, the current developments after the first project year, the expected results delivered at the project completion, and its impact on the long term.

During the BETTER project, challenges are introduced by promoters within the consortium addressing key societal areas, such as Food Security, Geospatial Intelligence (GEOINT) and Geohazards. A total of 27 challenges are proposed during three yearly challenge cycles, focused on problems such as detection of droughts, assessment of illegal crop cultivations or early warning of significant earthquakes, with an additional 9 brought by external promoters.

In the field of Food Security, the United Nations World Food Programme (WFP) is the main challenge promoter. Through the development of BETTER Pipelines, WFP expects to improve its preparedness in order to better address food security issues in humanitarian crises using Sentinels data, Copernicus Land, Atmosphere and Climate Change Services together with in-house datasets.

The three challenges addressed in first challenge cycle and currently under development are focused on delivering:
• Sentinel-1 Sigma-0 and coherence time series data for wetlands/water bodies and multi-temporal flood progression monitoring, extracting qualitative information on soil moisture or as an input for crop classification in specific areas of interest;
• Sentinel-2 and Landsat based vegetation indices and Sentinel-1 backscatter time series to extract information on the impact of WFP restoration activities (irrigation canals, land rehabilitation, dams, roads, etc.) on local environment and communities or the impact of conflicts on agricultural resources in hard-to-access areas (e.g. cropland abandonment, land degradation)
• Global vegetation, precipitation and land temperature time aggregated datasets and respective anomalies to a reference period to support the seasonal monitoring and early warning activities in all WFP regions.
Future challenges will diversify the range of data streams delivered to WFP focusing on: a) Multi-temporal EO for Humanitarian Operations; b) Hot-Spot EO Analysis Capacity for Natural Hazard Impact Assessment; c) Dynamic Land Cover Change Detection and Characterization.

The European Union Satellite Centre (SatCen) is the BETTER challenge promoter working in the Geospatial Intelligence (GEOINT) area. Their objective in the project is to enhance the capabilities of the Geospatial Intelligence community in the Space and Security domain through the provision of improved EO products and applications exploiting Big Data methods and techniques.

The first challenge cycle is focused on providing the following data streams for defined areas of interest:
• Sentinel-2 based multitemporal stack of thematic vegetation and water spectral indexes;
• Sentinel-1 multitemporal SLC and coherence stacks to detect changes over man-made and natural structures;
• DVI Maps, multitemporal stack of Sentinel-2 Bottom-of-Atmosphere reflectances and band color composites for the delineation of land cover types, with particular focus on vegetation.

In the two remaining years BETTER will implement other data streams related to: a) Change Detection and Characterization; b) Land Use / Land Cover; c) Thematic Indexes.

In the field of Geohazards BETTER will provide data streams to the Swiss Federal Institute of Technology Zurich (ETHZ) to derive information on volcanic activity, earthquakes, landslides and land subsidence to forecast and early warning of these natural disasters. Their research can be integrated in the scope of the Copernicus Emergency Management, mainly in the Risk and Recovery Mapping component. During the project these connections will be further analyzed to connect to additional promoters coming from these sectors and develop higher-level products that can bring additional value to the project.
The focus of this first cycle will be on the production of a global catalogue of:
• ENVISAT-ASAR interferograms (including coherence maps) before and during the event for earthquakes with a magnitude higher than 5;
• Sentinel-1 interferograms before and after the event for earthquakes with a magnitude higher than 5;
• Sentinel-1 co-seismic deformation change maps before the event and co-seismic;

Subsequent challenges will focus on: a) Forecasting the impact of earthquakes; b) Rapid generation of landslide inventories; c) Forecasting surface deformation.
The design, implementation and delivery of the Data Pipelines that answer the needs of the Data Challenges is driven by the definition of a set of pipeline requirements after the challenge definition process. Those requirements are extracted in a dedicated workshop in the beginning of each challenge cycle. After deriving this set of requirements, the BETTER development team composed by Terradue, Deimos and Fraunhofer design and implement the different pipeline components, following an Agile methodology. At each pipeline release, the promoters test and provide feedback to the development team, driving its improvements.

The collaborative development environment of the pipelines is based on a Jupyter Lab interface where several workflows archetypes (e.g. SNAP based archetypes, OrfeoToolbox -OTB- archetypes) are available for the development teams. At each release, the different software components will be uploaded to a software repository and enter a continuous integration/deployment environment where it will be merged into the full pipeline, tested with a validation dataset provided by the promoters, built, packaged, dockerized and made available in a production center. In this center it will start producing the data streams continuously and making them available to users for them to test and provide feedback. When a pipeline is finalized and answers all the requirements of the promoters, the challenge is considered achieved by the users.

After each pipeline implementation and challenge achievement, the main goal of the project is to promote the developed pipelines to other communities that might be interested in using those data streams.

In addition to communication, dissemination and engagement activities foreseen in the project there will be two hackathons open to everyone, one in the middle of the second challenge cycle and one in the middle of the third. In those hackathons, users will be asked to develop additional processing components on top of the current pipelines to extract information customized to their needs. This will allow to: a) showcase the use of the platform and demonstrate how the developed pipelines can support these wider communities, b) to identify additional micro-challenges by interested stakeholders who are interested in using the platform and variations of the pipelines to address their own user stories; c) get the feedback from the people participating in the hackathon on the robustness and usability of the developed pipelines.

Currently, BETTER is reaching the end of the first yearly challenge cycle with several pipelines already developed or under development until the end of this year. Those pipelines will be made available to other users in a First BETTER Hackathon to take place in mid 2019. In total, nine data Challenges are being addressed, involving 17 different EO input data streams and delivering to promoters 41 different output data streams (6 global or near global and 15 covering different regions around the world. To complement these data streams 7 new applications are being developed to perform user defined on-demand functionalities

The amount of Earth Observation (EO) data is increasing at an unprecedented rate. The European Copernicus program, in particular with Sentinel-1 and Sentinel-2 imagery in 10 m spatial resolution, is a major data contributor. These large volumes of data require not only sophisticated data storage and access options, but also new big spatial data analysis approaches.
The German Copernicus Data and Exploitation Platform (CODE-DE) is an infrastructure for EO data access, on-demand processing and value-added product generation. CODE-DE is operational since March 2017, hosts more than 1.300 registered users with more than 100.000 downloaded data products. The rolling archive host more than 650 TB of data with und 1.2 million data products. The catalogue client provides an enhanced solution to discover, view, and download available products. While a focus is set on data from the Copernicus program are other satellite imagery integrated into the portal.
The on-demand processing environment for high-performance cloud processing is operational since November 2017. The algorithms currently available are mainly based on the Sentinel tool box (SNAP) functionality, such as generic band math or dedicated time series algorithms. Processed data can optionally be mosaicked covering larger areas or a time periods. The capacity of the processing environment has been continuously exploited since March 2018 by selected applications, e.g. in the agricultural, maritime or snow and ice context, based on dedicated virtual machines or docker technology. This flexible and free processing approach supports different use cases – allowing developers, processing experts and data experts to work on the platform in support of their own processing scenarios.
The CODE-DE data and IT architecture will be explained and implemented user applications presented.

Over the last 16 years, DigitalGlobe high resolution imagery has been an indispensable source to extract geospatial information accurately and timely all over the world. Thanks to a world class constellation of high resolution imaging satellites, DigitalGlobe has constantly pushed the boundaries in providing extremely accurate data and new ways to see the planet we live in.
Our planet is changing at a rapid pace, creating challenges that have global impacts; understanding this change is essential if we want to save time, resources and even lives. Earth observation has a unique vantage point, providing the context and location intelligence required for the planning, use of and development from a plethora of interconnected devices and services.

What if you could not only see the world on a daily basis but also understand it? Geospatial Big Data converts millions of square kilometers of imagery to structured information that can be queried, indexed, and leveraged to provide deeper insight and understanding. Every day, DigitalGlobe collects over 4 million sqkm of high-resolution imagery, complete with super-spectral information. This creates an immense opportunity not only to see the globe but to understand it. Computer vision algorithms can locate every car, plane or baseball field, identify every building and map all the crops and minerals on the surface of the Earth. When the computers can’t figure it out, we turn to the crowd.

Today, the geospatial sector is undergoing some profound changes. The ability to rapidly analyse large amounts of data together with its increasing availability is bringing new users and creating innovative applications that few years ago were not feasible. Thanks to the power of the cloud, extensive experience in image processing and innovative machine learning approaches, users can now extract information at scale from geospatial data. During this presentation, I will explore some of the use cases that advanced machine learning and AI is enabling and demonstrate how space derived earth observation information can map and shape our understanding of the world through DigitalGlobe’s GBDX big data platform.