The Earth is a complex dynamic networked system. Machine learning, i.e. computer-based derivation of models from data, has already made important contributions to predict and understand components of the Earth system. For instance, classification of land cover types, modelling of land-atmosphere and ocean-atmosphere exchange of greenhouse gases, or detection of extreme events have greatly benefited from these approaches. Such data-driven information has already changed how Earth system models are evaluated and further developed. However, dynamic aspects of systems, such as the effects of spatial or temporal context (memory effects), still present substantial challenges for geoscientific prediction, change detection and classification by machine learning. Here we identify new developments in machine learning, in particular deep learning, that offer great potential to overcome these limitations in Earth system sciences, but also raise substantial new challenges. Promising near-future applications include nowcasting and short-term forecasting applications, as well as anomaly detection and classification based on spatial and temporal context information. As data-driven methodologies mature they can become important for applications such as seasonal forecasting, modeling of long-range spatial connections across multiple time-scales, and for uncovering yet unknown (tele)connections among variables. One key challenge and opportunity will be to integrate physical modeling approaches with machine learning into hybrid modeling approaches, which will combine physical consistency and machine learning versatility (cf. Fig 1).

The colour of water tells us a lot about the constituents of the water body and, where it can be seen, the floor of the sea of the lake. This comprises phytoplankton concentration, an indicator for biological activity in the water, important for local fishery, regional environmental monitoring and global carbon cycle, or suspended sediments which are highly relevant for shipping or transport of pollutants. Even the water depth influences the water colour and can be inferred. Water colour is measured by satellite spectrometers such as Sentinel 3 OLCI or Sentinel 2 MSI.

This richness and variety of information included in the water colour signal has a downside: the relationship between signal (light, or better radiance measured by the satellite) and causes (chlorophyll, sediment, CDOM, substrate, water depth, …) is governed by the radiative transfer equation (RTE) which is highly non-linear. The RTE is not analytically invertible, and the inversion is not unique. To make the situation even more challenging, the radiance measured by the sensor has to pass twice the atmosphere which adds additional unknown contributions to the signal. Several approaches have been developed to address this problem, and machine learning (ML) is one of them. ML can develop its strength in particular when it comes to optically complex waters, and in the Sentinel 3 ground segment processor of ESA / Eumetsat, a neural network based approach is used to perform the atmospheric correction, to derive the water constituent concentrations and also to provide the associated uncertainties.

This so-called C2RCC method is using forward RTE models for water and atmosphere to construct a training set for an ML based inversion. We tested different methods and concluded for a backpropagation feedforward neural network. The key to success is indeed not the neural net but the training dataset. The representativity for the natural conditions and its detailed statistical properties (ranges, correlation between variables) determine the quality of the NN to invert the RTE, but also the performance of the retrieval when validated with in-situ data.

The atmosphere poses another problem to ocean colour, and likewise to land-colour retrievals: the presence of clouds. While opaque clouds can be rather easily detected by physically based threshold tests (whiteness, brightness, height), semi-transparent clouds or sub-pixel cloudiness cannot be captured so easy by a universal algorithm. As part of the ocean colour processing a neural network, trained on manually selected individual pixels (PixBox approach), is delivering good results and is also used in the Sentinel 3 ESA/Eumetsat ground segment. We would like to stress here again that the construction of the training dataset is most important for the final quality of the cloud masking. An ML method can perform very well on its validation dataset; but if the population, from which training and validation dataset are taken, is not fully appropriate for the problem, the application of the ML method on real data and validation with in-situ data will deliver significantly degraded results.

In this presentation we will present the two examples of ML based algorithms with quite different approaches for the training dataset. Beside highlighting the challenges and showcasing the performance of the methods, including the innovative way to retrieve uncertainties, we will dwell into the importance of the definition of the population (training dataset).

Recent rapid developments in atmospheric composition sensors lead to huge opportunities for science, in particular for the Earth Observation (EO) technology for monitoring our planet from space. The new generation of Copernicus satellites, e.g. Sentinel-5 Precursor, provides a large amount of data, which has to be processed coping with the near-real-time requirements. Radiative transfer models (RTMs) are key components of the atmospheric processors, i.e. software specifically designed for the retrieval of atmospheric constituents from remote sensing data. The RTMs encompass our understanding of the physics behind the measurement process and relate the optical parameters of the atmosphere with the signal to be measured by the satellite. Today the requirements to the efficiency of RTMs are higher than ever. The traditional approaches implemented in the current atmospheric processors are inadequate to deal with so large amount of data. In fact, the amount of satellite data increases faster than the computational power. The remote sensing data is recognized as Big Data since it satisfies IBM’s 4V criterion: significant growth in the volume, velocity, and variety and veracity. New efficient techniques have to be developed for the next generation of atmospheric processors to cope with these high efficiency requirements. The design of new generation atmospheric processors can be regarded as a disruptive development. Artificial Intelligence (AI) is certainly one important way how to drastically improve the performance of RTMs, thereby enabling faster exploration of remote sensing Big Data. There is the consensus in the scientific community that AI, being a powerful tool, should not be used as a black box, but it should rather become an integral part of RTMs. Such a synergy between RTMs and AI establishes a new field of radiative transfer, namely AI4RTM, which represents a new physical basis for the AI4EO concept. Within AI4RTM, physics-based models are combined with the AI approach.
The following methods successfully treated by the machine learning approach boost the RTMs: classification, regression, dimensionality reduction and feature extraction. In our talk, all these concepts are outlined in the context of AI4RTM used both for forward modeling and retrieval.
AI has become a paramount tool in hyper-spectral data processing. Usually, the hyper-spectral data has a lot of inter-dependencies and correlations between different channels and spectral bands. In this regard, the dimensionality reduction techniques substantially reduce the number of RTM calls required for processing the data (del Águila et al., 2019). Such methods as the principal component analysis (PCA) integrated into RTMs provide a special kind of PCA-based RTMs, which can mimic the multiple scattering impact at low computational costs. We developed a novel hybrid PCA-based RTM. The simulations start with the simplified RTM (e.g. the two-stream model). By analyzing the set of hyper-spectral radiances, we identify a subset of wavelengths carrying the most relevant information and establish a projection of the initial data space onto a lower dimension subspace. In the reduced subspace, the optical properties are analyzed by means of AI and a correction function for the simplified RTM is found. Such hybrid implementation, in which AI treatment surrounds the RTM, improves the performance of hyper-spectral forward modeling by several orders of magnitude. This approach proved its efficiency for simulating the Hartley-Huggins band.
To accelerate the RTM computations and improve the efficiency of retrieval algorithms, artificial neural networks are used to efficiently parametrize the physical processes in the model. In this context, we use feed-forward neural networks to learn the behavior of a radiative transfer forward model by using a set of representative training data, whose input vectors are selected by means of the smart sampling technique (Loyola et al., 2016) to optimally cover the whole input space, and whose output vectors are the radiances returned by the radiative transfer forward model for every given input vector. Such kind of implementation has been used successfully as forward model in the retrieval of cloud properties from different sensors, including the newest TROPOMI (Loyola et al., 2018).
Regarding inverse problems, a special type of algorithms called Full-Physics Inverse Learning Machines (FPILM) (Efremenko et al., 2017; Xu et al., 2018) have been developed, specifically designed for the retrieval of atmospheric properties from remote sensing measurements. Unlike conventional inversion approaches, the FPILM, for instance, formulates the ozone profile shape retrieval as a classification problem, resulting in a reliable accuracy and significant speed-up. Its implementation consists of a training phase, wherein an inverse function is obtained from synthetic measurements generated using a RTM, and an operational phase in which the inverse function is applied to real measurements. In particular, the radiative transfer calculations are carried out once and offline. Another advantage of FPILM is its “simplicity”: the retrieval of the atmospheric properties of interest can be done very easily after the inverse function (with its coefficients) is determined from the training phase. Accordingly, the computational time of FPILM is several orders of magnitude faster than that of conventional approaches based on Tikhonov regularization with online RTM simulations, without degrading the retrieval quality. The FPILM algorithms are used for the operational processing of GOME-2 and now S5P/TROPOMI data.
As compared to the traditional retrieval techniques, our approaches exploit new computational paradigms and upgrade retrieval algorithms to a level, when they can be applied for analyzing Big Data. From different perspectives of understanding Big Data, there is always a risk to rely just on AI and to lose ability to interpret the results from the physical point of view. In our case, despite the fact that we are confronted with a sort of disruptive development based on AI, the physical component expressing our understanding of physical processes in the atmosphere is still preserved.


References

del Águila A., Efremenko D.S., Molina García V., Xu J. (2019): Analysis of Two Dimensionality Reduction Techniques for Fast Simulation of the Spectral Radiances in the Hartley-Huggins Band. Atmosphere, 10, 142.

Efremenko D.S., Loyola R. D.G., Hedelt P., Spurr R.J.D. (2017): Volcanic SO2 plume height retrieval from UV sensors using a full-physics inverse learning machine algorithm, International Journal of Remote Sensing, 38:sup1, 1.

Loyola D., Pedergnana M., Gimeno García S. (2016): Smart sampling and incremental function learning for very large high dimensional data, Neural Networks, 78, 75.

Loyola D.G., Gimeno García S., Lutz R., Argyrouli A., Romahn F., Spurr R.J.D., Pedergnana M., Doicu A., Molina García V., Schüssler O. (2018): The operational cloud retrieval algorithms from TROPOMI on board Sentinel-5 Precursor, Atmospheric Measurement Techniques, 11, 1, 409.

Xu J., Heue K.-P., Coldewey-Egbers M., Romahn F., Doicu A., Loyola D. (2018): Full-Physics Inverse Learning Machine for satellite remote sensing of ozone profile shapes and tropospheric columns. ISPRS. ISPRS Technical Commission III Symposium 2018, 7.-10. Mai 2018, Beijing, China.

Recent advances in deep learning have attracted great attention in remote sensing due to the high capability of deep networks (e.g., Convolutional Neural Networks (CNN), Recurrent Neural Networks, Generative Adversarial Networks) to model the high-level semantic content of RS images. To train such networks, a very large training set is needed with a high number of annotated images in order to learn effective models with several different parameters. To the best of our knowledge, publicly available RS image archives contain only a small number of annotated images and a large-scale benchmark archive does not yet exist. Thus, the lack of a large training set is an important bottleneck that prevents the use of deep learning in RS. In order to address this problem, fine-tuning deep networks pre-trained on large-scale computer vision archives (e.g., ImageNet) is considered in remote sensing community. However, such an approach has several limitations related to the differences on the characteristics of images between computer vision and remote sensing. Additionally, in the existing archives, remote sensing images are annotated by single high-level category labels that are related to the most significant content of the image. However, remote sensing images typically contain multiple classes and thus each image can be simultaneously associated with different land-cover class labels (i.e., multi-labels).
To overcome these problems we introduce a large-scale multi-label Sentinel-2 benchmark archive called as BigEarthNet, which consists of 590, 326 Sentinel-2 image patches, each of which is a section of i) 120x120 pixels for 10m bands; ii) 60x60 pixels for 20m bands; and iii) 20x20 pixels for 60m bands. Unlike most of the existing archives, each image patch is annotated by multiple land-cover classes (i.e., multi-labels) that are provided from the CORINE Land Cover database of the year 2018. The BigEarthNet is significantly larger than the existing archives in remote sensing and thus is much more convenient to be used as a training source in the context of deep learning. Experimental results obtained in the framework of Sentinel-2 image scene classification problems show that a shallow CNN architecture trained on the BigEarthNet provides much higher accuracy compared to a state-of-the-art CNN model pre-trained on the ImageNet (which is a very popular large-scale benchmark archive in computer vision). The BigEarthNet opens up promising directions to advance operational remote sensing applications and research in massive Sentinel-2 image archives.

In recent years, there has been an unprecedented increase in availability of Earth Observation satellites leading to unique opportunities for near real-time monitoring of spatio-temporal ecosystems processes. This unprecedented availability of different sensors creates challenges in terms of data integration as each mission is acquiring imagery at different temporal and spatial scales.

One of the most promising approaches to address the challenge of data integration is using Radiative Transfer Model (RTM) which are physical models that simulate the interaction of electromagnetic radiation with the leaf and canopy of vegetation. This approach is of interest because, in theory, it allows an estimate of biophysical parameters of vegetation independently of scale, time, and geographical location. Thus, it provides a unique opportunity for integration of remotely sensed data. The main issue of using these models, however, is that they are (i) computationally intensive and (ii) their performance is limited by the prior assumptions they require on the structure of vegetation.

Recently, machine learning models have shown the potential to address the computational challenges of these models by learning the relationship between biophysical parameters and simulated spectral responses of leaf and canopy. Machine learning based models also address the limits imposed by prior assumptions through using actual field collected data as ground truth that guides the machine learning estimates. Collecting ground truth data on vegetation parameters which is necessary for machine learning algorithms is, however, a very labour-intensive step and often not executed with the intent of being reused in other studies or geographical areas. Field spectroscopy measures the local spectral response of vegetation and can be used to provide a direct link between field spectra and the biophysical parameters. It, thus, provides generally usable ground truth data for validation of machine learning algorithms.

We intend to use machine learning in the context of nature conservation studies using remote sensing data. An extra open challenge that still needs to be addressed in these types of studies is modelling the uncertainty of unobserved natural driven processes (e.g. animal population and seasonal changes). The intention of our project is to advance the state of machine leaning algorithms to be robust against the uncertainties imposed by these natural processes.

In this talk, we will show our preliminary results in the experimental setup in the Oostvadersplassen nature reserve in the Netherlands. This reserve was known as one of the first examples of rewilding conservation that is currently under intense public pressure due to overpopulation of large herbivores (Heck cattle, Konik horses and Red deer). Our experiment consists of a data collection set up composed of multiple cameras which monitor the animal activities in the nature reserve, as well as, the phenology of the grassland. At the same time, we collect vegetation parameters as well as their field spectral responses. This data is then used to parameterize our machine leaning models to estimate the vegetation biophysical parameters from remote sensing data.

Our preliminary results demonstrate the potential of using machine learning methods for estimating grassland biophysical parameters and their sensitivity to the detection of varying grassland pressure in natural environment. Furthermore, they provide insights into the potential that advanced machine learning methods can bring into nature conservation as the continuous monitoring of the various agents influencing the nature reserve provide a near real time picture of the vegetation and animal population spatio-temporal dynamics.