Artificial Intelligence (AI) is expected to have enormous impact in addressing many of the greatest societal challenges, in particular regarding the environment. It is expected that it will help improve the quality of life of citizens. In addition, it will contribute greatly to increasing European industrial competitiveness across all sectors, including small and medium-sized enterprises and non-tech industries. All over Europe and within the European Commission, pan-european as well as individual governmental strategies and development plans have been set to stimulate and promote R&D for AI. For ESA and its Member States, AI is a spin-in technology considered as a game-changer that is gathering momentum also in space activities. It is an enabling technology for future missions and for operations and it an adds enormous value to increase scientific and value-added output from the variety of ESA’s space data.
The presentation will give an overview of ongoing AI activities across ESA.

Data preprocessing is often regarded as a necessity by many remote sensing scientists and practitioners. Extensive preprocessing, however, contradicts the notion of data-driven, end-to-end learning of models from the provided data. Further, the computational complexity of sophisticated preprocessing algorithms impedes a large-scale application of developed models.

Often, preprocessing is necessary when the data-inherent noise of the data, i.e., atmospheric perturbations and clouds, are not considered in the respective model.
We argue that recurrent neural networks are able to learn this type of data-noise in their model purely during training. Hence, we pursue a different strategy and explicitly avoid any preprocessing in our training and inference infrastructure. On our experiments, we observed that a straightforward gated recurrent neural network was able to achieve very similar classification accuracies in crop type mapping with preprocessed and not preprocessed data.

Gated recurrent networks process sequential data by multiple internal gates that define the paths of hidden states and gradients through time.
These gates control the degree of memorized, forgotten and utilized information at each time step given the current input and short- and long-term memory vectors from previous times.
They are heavily employed in natural language processing and have formed the basis of sentiment analysis and text generation from language models in the last years.
We adapted these networks for sequential satellite data [1,2] and observe, analogous to experiments in language domain [3], that some single internal recurrent cells trained to monitor specific states in the input data.
Given this methodical understanding, we hypothesized that some cells in these recurrent neural networks would be able to monitor the presence of clouds within a sequence of satellite images.

To test this assumption, we employed a convolutional recurrent neural network for crop type mapping on top-of-atmosphere imagery without prior cloud masking (attached as Figure 1).
Following the intuition of learning features and classifiers purely from the provided data, we observe with an ablation experiment (attached as Figure 3) that the model is able to produce equal classification accuracies on data with increasing levels of cloud coverage.
We further investigated this behavior qualitatively by visualizing internal states in the convolutional recurrent neural network (attached as Figure 2) which revealed that some neural cells have learned to identify clouds and were changing states on cloudy and non-cloudy imagery.

The ablation experiment and the qualitative visualizations indicate that deep neural networks are able to internalize a cloud-filtering mechanism while being trained on a different objective, i.e., crop type mapping.

These results show that state-of-the-art accuracies in crop type mapping [1] are possible without any sophisticated preprocessing techniques which allows a large-scale application of the model on European and global scales.

We did present initial results from this work at NeurIPS Spatio-temporal workshop [2] to a broader Machine Learning community and would like to take now the opportunity to present latest developments from this work to a combined Earth Observation and Artificial Intelligence community at the Phiweek conference.

References

[1] Rußwurm, M., & Körner, M. (2018). Multi-temporal land cover classification with sequential recurrent encoders. ISPRS International Journal of Geo-Information, 7(4), 129.
[2] Rußwurm, M., & Körner, M. (2018). Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery. NeurIPS 2018 workshop on Modeling and decision-making in the spatiotemporal domain.
[3] Karpathy, A. (2015). The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy blog, 21.

In this work, we introduce a Deep Learning (DL) method for noise removal in synthetic aperture radar (SAR) images. SAR systems provide high-resolution images of the Earth's surface and are a valuable source of information employed in a wide range of applications in the Earth Observation (EO) domain such as land cover classification, environment monitoring, military surveillance, and many others. Nevertheless, SAR images are contaminated by the speckle, i.e., the noise due to the constructive or destructive interference of multiple scatterers within the resolution cell of the radar. The result is a variation of the contrast of the pixels which are brighter or darker depending on the type of the interference. The goal is to remove the speckle from SAR images to improve their readability and to obtain noise-free data to support other tasks such as image segmentation or object detection.
Numerous approaches have been proposed in literature to this aim, such as, adaptive spatial-domain filters, wavelet-based filters, nonlocal filtering, and total variation denoising. However, these methods are characterized by the generation of artifacts in the reconstructed images, resulting in a poor performance in preserving the details of the original data. In addition, they are expensive from computational point of view. On the contrary, DL techniques already proved to be very effective in image processing, reaching high performance in a wide variety of Computer Vision (CV) tasks, e.g., image denoising, image classification, object detection, and semantic segmentation. Deep networks allow learning the non-linear function which maps input and corresponding targets, by adopting an ad hoc supervised optimization strategy. In the despeckling problem, the task is to learn a model that maps the elements within the set of speckled images to the elements of the noise-free ones. Thanks to the numerous EO missions, e.g., Sentinel-1, nowadays a huge amount of SAR images is available but it is not possible to obtain their clean counterparts for training. Alternative strategies need to be adopted to take advantage of DL technology.
In this context, our work aims at providing a new automatic tool to remove the speckle from SAR images, by overcoming the unavailability of training samples and increasing the performance with respect to state-of-the-art methods. In order to obtain pairs of input and target images, we have used a statistical prior about the speckle to generate the data synthetically. In particular, we resorted to the multiplicative model, i.e., the most commonly adopted for describing SAR images. For an amplitude image Y, we have Y=N·X, where X is speckle-free amplitude value and N is the speckle noise random variable that obeys to the Rayleigh distribution in the case of single-look image. We adopted this model to generate two kind of datasets. The first, the larger one, has been used as pre-training step and has been built using aerial images from the UC Merced dataset, originally created for land use classification. The original RGB images have been converted into grayscale images and then the noise has been added to each image by sampling from the Rayleigh distribution and applying the considered multiplicative model. The second dataset has been generated using averages of huge Sentinel-1 data stacks as clean starting images to which the speckle has been applied as for the first dataset. The goal of this second dataset is to adapt the model to the real domain resolution in fewer training steps. This process is known as fine-tuning.
The architecture used in our work is a modified version of U-Net, a convolutional neural network (CNN) originally introduced for biomedical image segmentation and adapted here for the SAR image despeckling task. Its symmetric encoder-decoder architecture ("U" shape) allows the understanding of the content of the input image by extracting relevant spatial features containing different levels of information as the network depth increases. An additional set of skip connections from the contracting path to the expansive one lets the local low-level information to be transmitted to the last layers and, at the same time, it helps the learning by solving the vanishing grandient problem.
The employed learning procedure belongs to the residual learning paradigm that proved to be effective in previous related work, i.e., SAR dilated residual network (SAR-DRN), which is the baseline for our experiments. Following this paradigm, the network is trained to extract the residual noise N = Y - X between the speckled input and the noise-free target. The loss minimized during the training is the mean-squared error (MSE) between the predicted residual noise and the target one.
In order to improve the quality of the reconstructed images we additionally introduced a total variation (TV) regularization, which encourages smoother results in homogeneous regions while preserving the structure of the original images.
We designed a huge amount of experiments to validate our approach on both synthetic and real SAR images. The obtained results show how the proposed approach achieves better performance than baseline methods and generalizes well also on data coming from different satellites, thus having different spatial resolutions. We also compared our performance with the DespecKS algorithm, demonstrating that our method is able to provide high-quality reconstructions without the need to resort to the temporal analysis over an entire SAR data stack but considering only a single image.

Machine learning techniques have been successfully introduced in the fields of Space Physics and Space Weather, yielding highly promising results in modeling and predicting many disparate aspects of the geospace environment. Magnetospheric ultra-low frequency (ULF) waves can have a strong impact on the dynamics of charged particles in the radiation belts, which can affect satellite operation. Here, we employ a method based on Fuzzy Artificial Neural Networks in order to detect ULF waves in the time series of the magnetic field measurements on board the low-Earth orbit CHAMP satellite. The outputs of the method are validated against a previously established, wavelet-based, spectral analysis tool, that was designed to perform the same task, and show encouragingly high scores in the detection and correct classification of these signals. Moreover, we present an application of this Artificial Intelligence technique to data from the Swarm mission. [Reference: G. Balasis et al. (2019), Journal of Space Weather Space Climate, 9, A13, https://doi.org/10.1051/swsc/2019010.]

When understanding the single polarization SAR images with deep learning, the texture features are usually learned automatically from the intensity. As an active microwave imaging, however, the complex Synthetic Aperture Radar (SAR) images not only contain the amplitude, but also the phase information, which is important and useful for interpretation. The time-frequency analysis (TFA) provides a physical understanding of the backscattering properties for each pixel in complex SAR images. As a consequence, a novel end-to-end deep learning framework to make the best use of both the physical properties of the objects and the spatial texture of the images is proposed. We start with a convolutional auto-encoder to learn the frequency features from each sub-spectrogram obtained by TFA, and then align them spatially. Next, the spatially aligned features in frequency domain and the low-level texture features obtained from a pre-trained SAR specific network in spatial domain are concatenated as the input of a post-processing residual network to learn spatial-frequency joint knowledge. The experiments were done on a large number of TerraSAR-X images. The proposed framework keeps the full information of complex-value SAR images, making a significant improvement compared with other spatial based deep learning methods in SAR image interpretation.
In order to learn the latent space that governs the backscatter values in SAR-imagery we explored the dimensionality reduction properties of variational auto-encoders (VAE). By taking both channels of the SAR data as input and mapping them to a compact, lower-dimensional representation, we constructed a single feature-vector consisting of the parameters of the latent space. This information was then fed to a classifier such as k-NN or SVM (Support Vector machine). Experiments on Sentinel-1 GRDH data using VV/VH polarizations showcased the capability of this method to extract the relevant features of the images, achieving an average precision/recall in the case of k-NN of 0.97 and 0.96, respectively.
Extracting physical scattering signatures from non-full-polarimetric images is of significant importance, but very challengeable. To achieve this goal and meanwhile exploring potentials of polarimetric SAR (PolSAR) images with different polarization modes and their combinations on this task, we proposed a contrastive regulated convolutional neural network (CNN) in complex domain. This method is to learn a physical-interpretable deep learning model from original scattering matrixes. The ground-truth is computed automatically by leveraging the Cloude and Pottier’s H-α division plane, which leads this work to an unsupervised learning mechanism. Considering the confused division boundary, a contrastive regulated term is computed in complex domain and added to the selected optimal loss function with a balancing trade-off coefficient. Experiments on DLR’s airborne, L-band F-SAR image demonstrate the feasibility of extracting physical scattering signatures from non-full-polarimetric SAR images. Moreover, the capabilities of different polarized images for achieving this are comprehensively analyzed and discussed.

A method to reconstruct missing information in satellite data using a convolutional autoencoder is presented. Satellite observations working in the optical and infrared bands are affected by clouds, which obscure part of the ocean underneath. A convolutional auto-encoder is developed to reconstruct the missing data based on the available cloud-free pixels in a long-term satellite time series. Training a neural network with incomplete data (or data with variable accuracy) is problematic. The present work shows a consistent approach using the satellite data and its expected error variance as input and provides the reconstructed field along with its expected error variance as output. The neural network is trained by maximizing likelihood of the observed value. The approach, called DINCAE (Data-Interpolating Convolutional Auto-Encoder) is applied to a relatively long time-series of Advanced Very High Resolution Radiometer (AVHRR) sea surface temperature data and compared to DINEOF (Data Interpolating Empirical Orthogonal Functions), a method to reconstruct missing data based on an EOF decomposition.

The cost function of the neural network is chosen such that the network provides the reconstruction but also the confidence of the reconstruction error (quantified by the expected error variance). An over- or underestimation of the expected error variance are both penalized by maximising the likelihood and assuming Gaussian distributed errors. This approach can be easily generalized to parametric probability distributions, in particular to log-normal distributions for concentrations like remote sensed chlorophyll concentration.


The presented reconstruction method DINCAE compared favourably to the widely used DINEOF reconstruction method which is based on a truncated EOF analysis. Formally there are similarities between an auto-encoder (composed by just 2 fully-connected layers) and an EOF projection followed by an EOF reconstruction. However, neural networks can represent non-linear relationships which is not possible with an EOF approach. Both methods were compared by cross-validation and the DINCAE method resulted in RMS error reduction from 0.4629°C to 0.3835°C.

The expected error for the reconstruction reflects well the areas covered by the satellite measurements as well as the areas with more intrinsic variability (like meanders of the Northern Current). The expected error predicted by the neural network provides a good indication of the accuracy of the reconstruction.

The accuracy of the reconstructed data under clouds was also assessed by comparing the results to in situ observations of the World Ocean Database 2018. Also compared to this dataset, the RMS error of the DINCAE reconstruction is lower than the corresponding results from DINEOF.

It is quite common that data analysis methods to reconstruct missing data tend to smooth the available observations in order to fill the area of missing observations. Therefore, the temporal variability (relative to the seasonal cycle) of the reconstructed sea surface temperature was computed from the original data and from the reconstructed data using DINCAE and DINEOF. The variability of the reconstructed SST with DINEOF generally underestimated the variability in the original dataset, but variability of the DINCAE reconstruction matched the variability of the original data quite well.