Artificial Intelligence is impacting all areas of Space science, across weather forecasting, disaster recovery, climate change, security, agrotech & much more. NVIDIA invests both in internal research and continual platform development to enable a diverse customer base. This brief talk will highlight recent work from NVIDIA Research, the Frontier Development Lab (collaboration with ESA) and academia, while offering a glimpse into state-of-the-art research.

Many applications in Earth Observation (EO) benefit from having reliable cloud masks, for example, land-use analysis or surface change detection. With the ever-increasing quantities of EO data being captured, the need for automated masking of cloud has become more and more pronounced. Moreover, the number and variety of instruments capturing this data has also increased, meaning general-purpose algorithms---which work on many instruments---are now very desirable. Traditionally, cloud masking has been carried out using physical thresholding techniques, which often require specific spectral combinations in the near and thermal infrared in order to be computable. However, Deep Learning has revolutionised many image processing tasks, and holds promise for doing the same in cloud masking, by using whichever spectral bands it has available as input. CloudFCN uses deep convolutional networks to segment input images into cloudy and non-cloudy regions. With a U-Net style architecture, spatial features from many scales are fused with pixel-level information for the model to use in classifying each pixel. We have previously demonstrated its state-of-the-art performance on Landsat 8, achieving accuracies above previously published algorithms. However, training (and validating) this model from scratch on a new product such as Sentinel-2 requires a significant amount of labelled data, which can be difficult to acquire. The current publicly available dataset for Sentinel-2 (Hollstein et al., 2016) is not of entire images, but rather of selected pixels from within images, which whilst useful for many pixel-based techniques, is unsuitable for training a convolutional network. Landsat 8 and Sentinel-2 do not have identical spectral bands, however many are comparable. In this study, we assess whether it is possible to use Transfer Learning techniques to minimise the amount of annotated data needed from Sentinel-2, by first training a model using the relevant bands in an existing Landsat 8 dataset, and then fine-tuning it with a small Sentinel-2 dataset. The primary outcome of this study is a high performance cloud masking algorithm for Sentinel-2. Further, this work will also inform the Sentinel-2 community as to whether the creation of a large, full-image labelled dataset would have utility in applying Deep Learning to the problem of cloud masking, or whether existing datasets from other instruments are sufficient.

There is an overwhelming amount of data from Copernicus satellites. For instance, thousands of images are produced every day representing a daily average of 3,45 TB of S1a/S1b data published. A significant amount covers ocean surface, used for a wide range of applications involving public and private stakeholders. Notwithstanding the current great impact of Sentinel-1 images for the oceanographic community, do we really exploit the full imaging capabilities of these C-band SAR data acquired over ocean surface?

Beyond wind field measurements, sea ice monitoring or oil spill detection, a set of metocean features are well observed by S-1 sensors. To name a few, atmospheric fronts, oceanic fronts, rain cells, micro convective cells, internal waves, gravity waves, biologic slicks, upwelling or wind streaks are phenomena of potential interest for many end-users while being generally discarded in the SAR images. For sciences academic and industries, new research perspectives and new potential applications/services could be opened if those mechanisms are automatically and systematically tagged. In addition, flagging of these features could also benefit to current products/services providing better data quality. With recent progress in computer vision thanks to deep learning approaches in conjunction with the rise of large database and higher computing power, these flagging activities are now possible.

Through the inspiring advances of Convolutional Neural Network (CNN) / Deep Learning (DL) applied to object recognition in image, we have implemented CNN approaches on Copernicus S-1 SAR images acquired over the open ocean. For the very first time (as far our knowledge goes – sea ice classification and iceberg vs vessel have been already investigated), DL frameworks are tested in the aim to provide a semantic segmentation of Sentinel-1 scenes acquired over open ocean surface, with the final objective to tag all these metocean features which have been neglected for years.

More specifically, a database of 37k Wave Mode (WM) imagettes has been annotated with 10 different classes (Pure Ocean Swell, wind streaks, micro convective cells, rain cells, biological slicks, sea ice, icebergs/bright targets, low wind area, atmospheric fronts, oceanic fronts). It is then used as training database in Inception V3 framework. Fine-tuned model provides cross-validation and test accuracy of more than 95%. To provide some insights on the classification results, T-distributed stochastic neighbor embedding (T-SNE) algorithm is used to project 10-dimensional predicted vector to a 2D map. This enables to highlight co-occurrence of phenomena, errors in the labelled database, and mixture/continuum of geophysical phenomena (for instance micro-convective cells / wind streaks). An assessment using an independant databse with 10k WM imagettes has been carried out, revealing good classification accuracy of more than 85% for most classes, with acceptable false alarms rate (below 10-20%).

As our initial training database provides only one label per imagette, we adopt a sliding window approach to attempt the semantic segmentation of oceanic SAR images. Sub-imagettes are extracted using either a random approach or a method based on the peak location of the Class Activation Map (Selvaraju et al. 2016) depending on whether the geophysical phenomena is localized or not on the original WM imagette.

In this presentation, we also discuss the potential of the methodology for large swath S-1 images (IW and EW modes).

Icebergs pose a significant threat to ships and maritime activities in ice infested waters, yet detailed information on iceberg locations is scarce. In the Antarctic, the size of an iceberg may vary over many orders of magnitudes and can be up to hundreds of kilometers across. The Arctic, however, has much smaller icebergs, with sizes in the order of the resolution cell of typical operating modes of synthetic aperture radar (SAR) systems. Common modes used over the Arctic include the Sentinel-1 Extra Wide swath (EW) mode with a resolution of around 90 m and the Radarsat-2 ScanSAR Wide mode with around 100 m resolution.

Ground range detected (GRD) images are by far the most commonly used product type for these modes because of their ease of use when it comes to visual analysis, the generally good software support, as well as their substantially smaller file sizes. This convenience comes at a cost, however, because the reduction in bandwidth from the single-look complex (SLC) samples to the multi-looked GRD significantly reduces the contrast of point targets present in the scenes.
In addition to the loss of contrast, the image statistics changes from a very simple one-parameter model for the single-look intensity (SLI) data to more complicated two (or even three) parameter models for the multi-looked products. Intuitively, the combined effects should be lower detection rates, increased runtime and lower discrimination power. It is therefore unfortunate that SLC’s are not consistently made available for the EW mode as they are for the interferometric wide swath (IW) mode.

In our study, we compare the performance of iceberg detection algorithms using SLC and GRD versions of the same scenes with two kinds of detection methods. We apply deep learning as a promising method to automatically learn hierarchical, representative and discriminative features from data. We then compare the deep learning detectors with classical constant false alarm rate (CFAR) inspired algorithms. For validation, we use manually detected icebergs from high resolution (10 m) Sentinel-2 optical data in addition to repeat observations using Sentinel-1 time series, where objects could be recognized based on their relative positions.

Our experiments show the value of having routinely access to the highest possible resolution data and we hope that this demonstration can nudge data providers to consistently make SLC versions of their data recordings available in the future.

Iceberg monitoring is required to ensure navigation and offshore operational safety in ice prone Arctic and sub-Arctic waters. The risk of iceberg collision with ships or offshore structures can be reduced by reliably detecting the icebergs well ahead of time so that safety measures can be taken. Satellite SAR imagery has been a reliable technology for operational iceberg monitoring in the Arctic for its all weather and all-season imaging capability. In SAR images, ship targets often become ambiguous with iceberg targets and extensive efforts have been made to reduce false alarms in the iceberg-ship detection algorithm. Use of machine learning to develop algorithms for distinguishing iceberg and vessel targets requires large validated data sets that are often costly, time consuming and, in some cases, inaccessible. Generating electromagnetic (EM) backscatter models of iceberg and ship targets can be a vital step in developing a robust iceberg/ship classification algorithm.
This paper presents the validation of an electromagnetic (EM) backscatter model of icebergs at C-band by comparing the performances of target classifiers trained with both modelled and real SAR data. Recently, an EM model was developed for lower incident angles and compared with SAR signatures and radiometric properties (i.e., total radar cross section (TRCS)) of dual-pol Sentinel-1A data [1]. Once quad-pol RADARSAT-2 images were available for icebergs, an EM model was developed to model iceberg backscatter at higher incident angles with a full polarimetric capability [2]. Polarimetric SAR data provided an opportunity to compare EM model of iceberg in terms of polarimetric decomposition, in addition to SAR signatures and TRCS. Finally, EM backscatter modelling of iceberg was tested in the context of iceberg/ship classification by comparing classifiers performance with simulated and real SAR image data of icebergs. This step is critical to check possible simulation errors (i.e., differences between the simulations and the real SAR data) could affect the integrity of the classifiers.
Simulated SAR image data were obtained in combinations of imaging beam modes and scene parameters to produce a total of 216 simulated Sentinel-1 C-band SAR images. Parameters consisted of Sentinel-1 IW1 (33.1˚) and IW3 (43.1˚) beam modes with varying wind speed (5 and 10 m/s), wind direction (0˚, 45˚ and 90˚) and target orientation (0˚, 45˚ and 90˚). Simulations were created from a SAR simulator called GRECOSAR, which took three-dimensional (3D) profiles of iceberg and ship targets and parameters necessary to closely mimic the real scenes. 3D models of three icebergs were captured in a field study off the coast of Bonavista, Newfoundland and Labrador, Canada in June 2017. Three generic ship models were sourced from an online inventory and scaled to a size equivalent to that of the iceberg targets. Real SAR image data were drawn from in house dataset collected from a complementary research program. Classifiers, including Support Vector Machine (SVM), Random Forest (RanFor), k-Nearest Neighbour (kNN) and Neural Network (NN) were trained with targets from modelled SAR data and then gradual mixed with real SAR data. Target classifier performance from the modelled target data was shown to be similar to classifiers trained entirely from real SAR data. The similarity in accuracy provides an indication of the validity of the modelled SAR data for this specific application.
References
[1] M. S. Ferdous et al, "A Comparison of Numerically Modelled Iceberg Backscatter Signatures with Sentinel-1 C-Band Synthetic Aperture Radar Acquisitions," Canadian Journal of Remote Sensing, pp. 1-11, 2018.
[2] M. S. Ferdous et al, "An Iceberg Electromagnetic Backscatter Model: Comparison with RADARSAT-2 Image Data, Evaluation of SAR Signature Variability with Ocean Parameters and a Mathematical Model ", Under review.

Synthetic Aperture Radar (SAR) satellite images are used extensively for producing sea ice charts in support for Arctic navigation. However, due to ambiguities in the relationship between C-band SAR backscatter and ice conditions (different ice types and concentrations as well as different wind conditions have the same backscatter signature) the process of producing ice charts is done by manual interpretation of the satellite data. The process is labor intensive and time consuming, and thus, the amount of charts that are produced on a given day is limited.
Automatically generated high resolution sea ice maps have the potential to increase the use of satellite imagery in ice charting by providing more products and at shorter delays between acquisition and product availability.
In this work we explore data fusion and image segmentation techniques with Convolutional Neural Networks to produce per pixel predictions from Sentinel 1 (S1) SAR images and AMSR2 microwave radiometer measurements of Ice/water. The work is carried out under the Danish Automated Sea Ice Products (ASIP) project in a collaboration between the Danish Meteorological Institute and the Technical University of Denmark. The aim is automating a substantial part of the current manual process of providing artic marine users with ice information.
For the study a dataset of more than 900 ice charts and corresponding Sentinel-1 SAR imagery has been collected. The core of our algorithm consists of a Convolutional Neural Network that models image features at different scales by the use of dilated convolutional filters. The architecture of the algorithm further allows us to merge S1 images with AMSR2 measurements in a data fusion approach that exploits the best properties of each measurement. While the 40m pixel size in Sentinel-1 data enables extraction of ice information at an unprecedented high resolution, the AMSR2 measurements contributes with a high contrast between ice and water independent of wind conditions. Future studies in the project will investigate the importance of additional meta data in the ice prediction, such as weather information, sensor viewing angles, geographic location, etc.