The ongoing rise in missions to observe Earth from space, especially the various Copernicus’ Sentinel systems not only increases the volume of data daily, but also contributes to the variety of data, the velocity of data availability and its veracity. Considering the availability of existing extensive archives of Earth Observation (EO) data, e.g. ENVISAT (MIPAS, SCIAMACHY), as well as various future missions, e.g. EarthCARE (MSI), Sentinel-5 A (UVNS), Metop-SG, the field of EO has yet reached an era of Big Data. These circumstances generate challenges regarding analysis and processing which ask for innovative services and solutions.
The discipline of atmospheric sciences poses an additional difficulty in efficiently accessing and analysing all available data: the variety is high as the source of atmospheric data is threefold with data coming from EO systems, models as well as in-situ measurement. The heterogeneity and multidimensionality of the so-called data triangle (EO, model and in-situ data) makes an efficient exploitation of the full potential of the available information more challenging.
Initially a demonstration platform (Technology and Atmospheric Mission Platform (TAMP)) to show atmospheric science users the possibility to access, visualize, process and download heterogeneous, multidimensional data, the concept of the Virtual Research Environment (VRE) was further developed through VEEDAM (Virtual Exploitation Environment Demonstration for Atmospheric Missions), which involved a larger user-base, more data sources and processing functionalities and the possibility to make use of a Jupyter notebook interface. Being the second evolution of the platform, TOP aims at reaching an operational and commercially viable status through enlarging the data offer to all Copernicus atmospheric and climate data products (CAMS, C3S, Sentinel 4, 5, 5p) while allowing dynamic computational resources allocation.
For TOP to achieve largest reach, the deployment on a Copernicus Data and Information Access Service (DIAS) is foreseen. Deployed on a DIAS, TOP would be the first operational platform implementing the data triangle and hence create an atmospheric multi-source data cube, stimulating a multidisciplinary scientific approach, due to the availability of various collections. Data access barriers would be broken down: based on OGC-standardised data discovery and access, multiple heterogeneous datasets can be explored simultaneously, without the need to have any knowledge of the original products. Moreover, as (new) data available on the DIAS will be directly accessible through TOP, the time to collect and prepare data would be omitted. As a consequence, also local storage space and personal computer resources would be saved, since all data will reside in the remote environment (DIAS) and the processing can be performed remotely.

The availability of Copernicus Sentinel-1 data creates a luxury problem for the satellite radar interferometry (InSAR) community. Thanks to the Copernicus open data policy, and the Sentinel-1 mission characteristics, a voluminous amount of information and data is generated and stored. In communication theory, information needs to be processed through own experience and become useful knowledge before being actionable. In EO, a similar process will have to take place, with the satellite data processed through algorithm to generate actionable products for the end users. More specifically, in the case of large-scale ground deformation products, an additional challenge arises.

Even if we assume that three-Vs (Volume, Velocity, Variety) big data challenge of Copernicus Sentinel-1 data is solved, by allowing unlimited computing resources and solving the challenges related to production and scaling of the ground deformation product, the sheer volume of time-series information being generated makes it not easily actionable for the user. Therefore an additional layer of (post-)processing is required. The proper understanding of the nature of this additional layer could be crucial to the success of any service building on the Copernicus Sentinel-1 constellation data. In particular, it will sometimes be necessary to derive simplified higher-order products specifically tailored for downstream users.

In this contribution we will discuss and analyze the questions arising after production of a large scale deformation map. The challenge that will be discussed revolves around the process of unfolding the added value of a very large and complex dataset for a diverse audience of end users. Specifically, an in-house developed scalable web service for exploration and exploitation of large-scale ground deformation products will serve as a platform for the discussion. We will present examples and open challenges for successful analysis of large amounts of ground motion data, such as spotting trends, identifying correlations and unexpected relationships, and explore multiple hypotheses.

ABSTRACT

The ESA-NASA multi-Mission Algorithm and Analysis Platform (MAAP) is dedicated to the BIOMASS [1], NISAR [2] and GEDI [3] missions. This analysis platform will be a virtual open and collaborative environment. The main goal is to bring together data centres (Earth Observation and non-Earth Observation data), computing resources and hosted processing in order to better address the needs of scientists and federate the scientific community.
The MAAP will provide functions to access data and metadata from different sources such as Earth observation satellites data from science missions; visualisation functions to display the results of the system processing (trends, graphs, maps ...) and results of statistic and analysis tools; collaborative functions to share data, algorithms, ideas between the MAAP users; processing functions including development environments and an orchestration system allowing to create and run processing chains from official algorithms.
Currently, the MAAP is in its pilot phase. The architecture for the MAAP pilot foresees two independent elements, one developed by ESA, one developed by NASA, unified by a common user entry point. Both elements will be deployed on Cloud infrastructures. Interoperability between the elements is envisaged for data discovery, data access and identity and access management.
The ESA element architecture is based on technical solutions including: Microservices, Docker images, Kubernetes; Cloud-based development environments (such as Jupyter or Eclipse CHE) for the MAAP algorithm developers; a framework to create, run and monitor chains of algorithms containerised as docker images. Interoperability between both ESA and NASA elements will be based on CMR (NASA Common Metadata Repository) and secured with the OAUTH2 protocol.
This presentation focuses on the pilot platform and presents the overarching joint architecture and how interoperability between the NASA and ESA elements will be achieved. It also gives insight into the architecture of the ESA element and its technical implementation.


REFERENCES

[1] T. Le Toan, S. Quegan, M. Davidson, H. Balzter, P. Paillou, K. Papathanassiou, S. Plummer, F. Rocca, S. Saatchi, H. Shugart and L. Ulander, “The BIOMASS Mission: Mapping global forest biomass to better understand the terrestrial carbon cycle”, Remote Sensing of Environment, Vol. 115, No. 11, pp. 2850-2860, June 2011.
[2] P.A. Rosen, S. Hensley, S. Shaffer, L. Veilleux, M. Chakraborty, T. Misra, R. Bhan, V. Raju Sagi and R. Satish, "The NASA-ISRO SAR mission - An international space partnership for science and societal benefit", IEEE Radar Conference (RadarCon), pp. 1610-1613, 10-15 May 2015.
[3] https://science.nasa.gov/missions/gedi

Human society increasingly depends on information derived from Earth Observation data. In particular, there is an growing demand for information facilitating agriculture, forestry, and hydrology. To this purpose the number of space-borne satellites and small cube-sat constellations are projected to increase dramatically over the next years. This increase provides enormous challenges in terms of retrieving actual information from this big data. The current general trend is to create individual land surface products for each satellite mission. This approach originates from the fact that satellites are usually has singular research-mission, that require only a specific number of land surface products to be created. As such, the majority of available products are created using a bespoke single-sensor approaches. This production-heterogeneity severely limits the advancement of research fields due to inconsistencies in the combinations of such land surface variable.
Land surface parameter retrieval suffers from illposedness: the fact that there are fewer observables than the amount of desired parameters (required for accurate retrieval). By only considering observations from singular satellites, this illposedness is worsened especially considering that varying spectral-sensitivities of different land surface parameters. A multi-sensor approach capable of integrating such sensitivities, by accurately modelling the physical radiative processes, resolves these limitations. Furthermore, such a multi-sensor approach also allows to benefit from synergies of using multi-scale/heterogeneous observation types/varying temporal frequencies of different sensors.

In this view, within the MULTIscale SENTINEL land surface information retrieval PLatform (MULTIPLY) project, the main objective was to create a framework to integrate observations from different sensors in order to obtain the best possible estimate of the land surface state, taking into account the different characteristics of different sensors and data streams. The MULTIPLY framework is developed from the original idea’s implemented in the EOLDAS system but with emphasis on operationally, enhanced consistency (across sensor types) and improved gap-filling. Specifically: MULTIPLY aims to
1. combine data from SAR observations with optical remote sensing data using compatible radiative transfer models
2. design a data assimilation platform incorporating as many information sources (observational, prior, multi-temporal) to optimally retrieve satellite information that are gap-free.
3. deliver a set of internally consistent data products at different resolutions (coarse and high) with quantified uncertainties.
4. Explore potential applications that demand consistent land surface products, such as agriculture, forestry and soil moisture.
At present, the prototype of the MULTIPLY has been implemented as a cloud-service and is currently undergoing expert-user-evaluations.

During the presentation we present the operational MULTIPLY framework (http://www.multiply-h2020.eu/), the detailed methodology, as well as the results of our evaluation activities. In the presentation, specific emphasis will be put on the consistent coupling of Sentinel-1 (SAR) observations together with Sentinel 1&3 (optical) observations (together with auxiliary ecological information) in order to jointly and consistently retrieve soil parameters as well as vegetation traits and b) elaborate on the results of the fieldwork validation efforts.

Global cloud free mosaics are providing background layers supporting the rendering of other information layers. They can also be directly used as a basis for extracting new information layers.

With over 3 TiB of Sentinel-2 (A and B) products published per day, the creation of a global cloud free Sentinel-2 mosaic from a given reference year is challenging in terms of image selection and computational needs. These challenges are addressed in this paper by adopting a multiresolution approach based on optimal selection procedure following the method described in [Kempeneers and Soille, 2017]. This approach is effective in the sense that it minimises the number of images contributing to the creation of the global mosaic and efficient in the sense that the selection is based on the processing of images at much coarser resolution than the native imagery. The selection and subsequent compositing is also suitable to high-throughput computing for it can be applied in parallel to all 35,000 100 x 100 km tiles of Military Grid Reference System (MGRS).

The optimal selection is an iterative process that requires the quicklooks of the full resolution imagery as well as their associated vector cloud masks. At each iteration, the image maximising a given objective function was selected in order to obtain the minimum number of images required for a cloud free composite. The maximum number of overlapping images was constrained to 5. As a result, less than 5% of the available Sentinel-2 images in 2017 were selected. On average, for each MGRS tile, 3 overlapping images were sufficient to generate a cloud free global mosaic (see Fig. 1).

The images selected on the basis of their quicklooks were then downloaded at full spatial and spectral resolution (Level 1C). These images were then atmospherically corrected using the Sen2Cor L2A processor (version 2.5.5; European Space Agency) to produce the Level 2A images. A composite was then calculated for each tile for both the levels 1C and 2A, based on the maximum normalized difference vegetation index (NDVI) for each pixel (Fig. 2). The tiles are then mosaiced for each individual UTM zone.

The whole processing benefited from the batch processing capabilities of the JRC Big Data Platorm (JEODPP) [Soille et al., 2018]. With 1,000 cores available for processing, the total processing time took less than 4 days. The mosaic can also be viewed using the JEODPP interactive analysis and visualisation based on Jupyter notebooks [De Marchi et al., 2017, De Marchi and Soille, 2018] and shared thanks to its html export function [https://cidportal.jrc.ec.europa.eu/services/shared/html/S2Mosaic2017.html] . The resulting mosaic is being used for generating the 2017 Global Human Settlement Layer.

References

D. De Marchi and P. Soille. Big geospatial data visualization and analysis in Jupyter. Presentation at the EuroSciPy 2018 conference, Trento, Italy, September
2018.
D. De Marchi, A. Burger, P. Kempeneers, and P. Soille. Interactive visualisation and analysis of geospatial data with Jupyter. In Proc. of the BiDS’17, pages 71–74, 2017.
P. Kempeneers and P. Soille. Optimizing Sentinel-2 image selection in a big data context. Big Earth Data, 1(1–2):145–158, 2017.
P. Soille, A. Burger, D. De Marchi, P. Kempeneers, D. Rodriguez, V. Syrris, and V. Vasilev. A versatile data-intensive computing platform for information retrieval from big geospatial data. Future Generation Computer Systems,
81(4):30–40, April 2018.

The Forestry TEP platform is in transition to an operational phase in 2018, after having completed the main development project funded by the European Space Agency ESA. The activities in this period include migration to a DIAS infrastructure, on-boarding a number of service provider companies, expanding the user base, enhancing the operational features, and improving the overall user experience.

Forestry TEP enables efficient exploitation of optical and radar data from Copernicus Sentinels and other Earth Observation sources for forest analysis and monitoring. Commercial, governmental and research users globally can access processing services and tools, collocated with the data, for generating value-added forest information products. Via the platform, the users can also create and share their own processing services, tools and generated products. The platform functions as a worldwide marketing channel to commercial remote sensing services and a forum for networking.

A selection of open-source processing services is available on the platform, enabling generation of vegetation indices, mapping of land cover, analysing forest change, estimating forest biomass, and performing pre- and post-processing tasks for satellite data. Additionally, the users can utilize a selection of popular software tools, directly on the platform through the web browser.

Developers in academia or value-adding industry are able to bring their existing or novel methods as new processing services on the platform, with the help of a web-based development interface. Developers can share the services to defined groups of colleagues or make them public to all users, for free or for licensed use. Service deployment is based on the Docker technology, in a way that frees the user from performing any software installations. Developers can base their methods on any programming language available on the Linux environment, such as Python, and the libraries of e.g. Orfeo Toolbox, SNAP and GDAL.

During the current transition of Forestry TEP to an operational phase, in particular focus is supporting a number of value-adding companies to on-board the platform with their specific use cases. Concurrently, development of the platform continues based on identified user needs, to improve the operational features and overall user experience. Planned developments include, for example, supporting delivery of output products to external systems via standard interfaces, enabling automatic selection of data for processing services, improving management of user’s own data on the platform, more efficient processing of large datasets, and capability for setting up automated processing.

With continued partial support by ESA, the platform is now operated jointly by VTT Technical Research Centre of Finland Ltd as the coordinator, forestry domain specialist and manager of user communications, and CGI IT UK Ltd as the system developer and integrator. The platform is preparing transition to the infrastructure and data access services of a Copernicus Data and Information Access Services (DIAS) provider by end of 2018.

Forestry TEP is currently a key asset in several H2020 projects, including the Big Data PPP project DataBio. The platform is available for further collaboration in both research and value-adding projects.