Urban environments are complex and fast evolving over time due to the frequent interaction between humans and the natural system. Timely and accurate change information in urban areas is essential for successful urban planning and management and crucial for decision-making related to sustainable development. Change detection of urban environment is considered a mature research field, mainly because has been extensively studied from multidisciplinary scientific teams. However, the change detection problem from multi-temporal earth observation data still remains a challenge mainly due to the continuously increasing needs of the society and the evolving requirements of the stakeholders.
Two different approaches, depending on how using multi-temporal earth observation data (i.e. images), can be used to address the change detection problem. The first one (one-shot approach) consists of using directly the difference between two images, acquired in the same area but in a different time; while the second one (using two stages) performs change detection after a classification step. The use of this latter, allows to create a binary mask that is able to describe building (or no-building) in an image; then in order to perform change detection, the two masks obtained for each images have to be subtracted. According to this, more accurate are the results of the classification better is the detection of some changing. Then, due to the importance of the classification step, the main goal of this work is to classify building using two VHR satellite images acquired with difference in time. Each image is classified using non-parametric classifiers such as Random Forest (RF), k-Nearest Neighbor (kNN), and Support Vector Machine (SVM) as well as Deep Learning approaches. Comparison between these methods have been performed on different data, using layers information of Open Street Map as training samples and manually detected area as validation data.
According to experimental results in images, acquired in different time steps, capabilities and potential use of the Deep Learning approaches will be analyzed in order to understand how these techniques can contribute in the comprehension of this topic.

Area C - Board 323

Object detection in very high resolution (VHR) optical remote sensing images is a fundamental and important task in the field of aerial and satellite image analysis. The object detection is widely used in environmental monitoring, urban planning, traffic planning, etc., and has attracted considerable research attention. With the development of machine learning, especially deep learning, object detection can be regarded as classification and regression problems. The deep-learning-based methods have achieved significant improvements for detecting some specific objects due to the powerful ability of learning high-level feature representations. Among various deep learning methods, the faster region-based convolutional neural network (R-CNN) is a representative two-stage detector. The faster R-CNN first uses region proposal network (RPN) to predict the region proposals, and the features extracted from the proposals are then applied to locate and classify the objects. In recent research, the methods that are proposed based on faster R-CNN have achieved good performance on object detection.
However, in the methods based on faster R-CNN, there are still some issues need to resolve. (a) The backbone network for faster R-CNN plays a key role in feature extraction, however, the backbone networks in most of the detection tasks employ the models which are designed for image classification, such as VGGNet and ResNet. These models are pre-trained using ImageNet dataset and then fine-tuned in the detection process. Although object detection can be regarded as a classification problem, there are still some differences between them. Object detection not only needs to predict the categories of the objects, but also needs to locate their corresponding positions. Therefore, some designs for classification tasks are not suitable for object detection, such as large down-sampling scale.
(b) The faster R-CNN predicts the positions and categories of the objects based on the proposed regions. However, this method ignores the information of the scene. It is well known that scene is helpful in distinguishing different objects, especially the objects with a similar appearance. For instance, the bridges look the same as roads in satellite images, and can even be regarded as a part of roads. In fact, it is very hard to distinguish them when only region features are used.
To address these problems, we propose an object detection method based on faster R-CNN. The framework has two characteristics. (a) the backbone network adopted is more suitable for the detection task, for example, it can maintain the spatial resolution. (b) the region feature is combined with the scene feature to better discriminate different types of objects.
Our major contributions of this paper are summarized as follows.
(a)We introduce the DetNet as the backbone network in our framework, which is a novel network specifically designed for object detection. DetNet maintains high spatial resolution in deeper layers, which can help to locate the boundary of large objects and to find the missing small objects. DetNet also adopts a dilated bottleneck with 1×1 convolution projection at the beginning of every stage. So it can increase the divergence between input and output feature maps and create a new semantic stage when spatial size is unchanged. Meanwhile, DetNet can be pre-trained on ImageNet dataset, as in traditional classification networks.
(b)Our framework tries to extract the features of the scene and combines them with the region features. First, the same backbone network is used to extract the scene features and region features. To reduce the dimension of scene features, we then pool the scene features and region features into the same size. They are both mapped to feature vectors by fully connected layers. Last, two feature vectors are concatenated and the fused features are used to predict the labels and positions.
In order to evaluate the performance of the proposed method, a public multi-class geospatial object detection dataset, NWPU VHR-10 dataset, is employed. The dataset contains ten classes of objects. There are 650 images in positive image set of the dataset, with each image containing at least one object of given classes. All the images are manually annotated by experts.
The evaluation metrics adopted in the experiments are precision-recall curve (PRC) and average precision (AP). The precision metric measures the proportion of true positives in all detection results and the recall metric measures the proportion of the correctly identified positives in all positives. The AP computes the average value of precision over the interval from recall=0 to recall=1. In practice, we compute the mean AP (mAP) for all classes to evaluate the overall performance of multi-class detection.
In the first experiment, we compare DetNet with some widely used backbone networks, such as ResNet. In the results, the mAPs of ResNet-50, ResNet-50 with FPN and DetNet-59 are 78.34%, 80.31% and 83.40%, respectively. The results show that the new backbone network can improve the detection performance of faster R-CNN.
In the second experiment, we adopt DetNet-59 as the backbone network, the detection framework combines region features and scene features. For comparison, the original DetNet-59 is also tested, namely the framework that only use the region features. The results show that the scene features contribute a mAP improvement of 2.1% (from 83.40% to 85.50%) to the DetNet-59. Especially, it raises the AP by 16.76% on the detection of bridges. The results demonstrate that combining scene features with region features can detect more objects than only using region features.

Area C - Board 324

Machine Learning has been used extensively in remote identification tasks. In recent years it has been solving more sophisticated image classification, natural language processing problems using hierarchical models called Deep Learning. However, the success of machine learning in scientific and engineering applications has been tempered by the issues of interpretability or the “black box” nature of its solutions. Deep Learning has been shown to behave in surprising ways when applied to certain types of synthetic data called adversarial examples that fool classifiers with high confidence. This makes its use questionable for critical applications or decision making, such as in meteorology and for a better understanding of Earth’s dynamics. This project explores the issues of incorporating the relevant physics, chemistry, biology, or other natural sciences into the Machine Learning / Deep Learning framework to help mitigate the concerns of scientists.

Models of data can have different fundamental structures and varying complexity. Some model structures attempt to capture the causal relationship of their constitutive variables, as is often the case in the natural sciences while other models only seek the identification of correlation and association amongst variables as in classical statistics. Typically models are constructed from parameterized functions. The number of functions and parameters is the model complexity (shown on the horizontal axis of the figure below). A model-driven strategy uses simple models with limited degrees of freedom often based on known scientific principles. In contrast, a data-driven approach uses a large number of functions and parameters that can scale with the amount of data. These models may not be directly related to scientific models but only seek to make reliable predictions, y, about new data, x.

It will be argued that Bayesian inference provides a modular architecture for incorporating physics in the forward models used in science into a probabilistic machine learning strategy that can both guide the experimental design and analyze the resulting data. In particular the role of “labels” in supervised learning is related to the utility of retrieved properties in remote sensing within a Bayesian form of probabilistic machine learning. Decision retrievals represent an interplay between the inference of physical properties as measured by remote sensing and the requirements of decision-making based on those properties.

Area C - Board 325

Satellite images have become abundant recently, providing a high-resolution view of the total surface of the earth. By analyzing the satellite imagery, we can understand multitude of processes such as measuring the rate of construction in a country related to economic growth, measuring the oil tanks, planning and organizing the traffic, calculating the deforestation and its effects, helping wildlife preservation by counting animals and tracking their movements, discovering archaeological sites, mapping the damaged regions due to a natural disaster, and many more. And for all of these problems, the solution starts with the labelling of satellite images.

Recent advancements in the Deep Learning (DL) field have enabled the automatic labelling of images for tasks such as classification, target detection and semantic segmentation, often surpassing human-level accuracy at much lower cost. This issues are of special importance in satellite imagery, where an EO expert can spend up to a week to label a single image. Despite the clear advantage than labelling algorithms based on DL techniques provide nowadays, there is still some drawbacks. The quality of the algorithms heavily depends on the availability of big labeled datasets and require considerable computing resources. Concerning computational resources, the spread use of powerful graphics cards and cloud computing platforms enable the training of large networks. On the other hand, the availability of quality labelled datasets is still short. Some examples of datasets can be found at the AIRBUS One Atlas Sandbox portal where labeled images for ship, plane and car detection are available for machine learning (ML) applications.

In the POINTOUT project funded by ESA through the EO SCIENCE FOR SOCIETY PERMANENTLY OPEN CALL, our objective is to develop a platform to build and share collaborative training datasets for the combined EO/DL communities. Also, to implement generic DL algorithms for target detection in EO scenes for expert and/or non-expert users online. Tools will be provided to pin out and label specific objects in an EO scene through collaborative effort to establish large training datasets of localised objects. With these datasets in place, automatic online training of Neural Networks (NNs) over areas of interest will be enabled. A powerful application of such a platform is the supervised pseudo-labelling of images, where already trained models label images that are then validated (or corrected) by users, dramatically accelerating the process. Another approach that this platform allows is the on-line training of models. This means, the continuous training of already working models as soon as new data is available. NNs usually behave differently depending on the nature of the data. Satellite imagery present some peculiarities, such as the availability of more than three (RGB) channels, multi-spectrum information, very large images with high-resolution, etc. It will be our focus to select and implement the models and optimization techniques that best fit this characteristics.

Several entities have already expressed their interest in a platform like this: GMV provides services in the security markets with interest in detecting temporal structures such as vehicles, camps and military equipment. Sacyr is a construction company looking for solutions to monitor remotely construction sites and heavy equipment. Generalitat de Catalunya, as a public organisation is interested in monitoring legal constructions such as swimming pools. Summarizing, providing a tool for easy and collaborative labelling will enable non-experts users to have access to these powerful tools. And, as more data is available, the better the accuracy of the models will be. This is a win-win scenario for both EO and ML scientists.

Area C - Board 326

Modern Earth Observation (EO) systems produce huge volumes of data every day, involving programs that provide satellite images at high spatial resolution with high temporal revisit period. High-resolution Satellite Image Time Series (SITS) represent a practical way to organize this information, which is particularly useful for area monitoring tasks. A notable example is the Sentinel-2 mission, which is part of the Copernicus Programme, i.e., a programme developed by the European Space Agency (ESA) that involves a constellation of satellites monitoring different aspects of the Earth surface. The Sentinel-2 mission allows monitoring the entire Earth Surface at 10m of spatial resolution with a revisit period between 5 and 10 days, supplying optical information ranging from visible to near and medium infrared. One of the main advantages of this mission is that the produced data are publicly available.
For these reasons, the use of SITS is gaining increasing success in a plethora of different domains, such as ecology, agriculture, mobility, health, risk monitoring and land management planning. The use of SITS has been proved beneficial in the context of Land Use/Land Cover (LULC) map generation, thanks to the possibility to exploit temporal features to better discriminate among classes that exhibit different temporal behaviors, i.e., with respect to the results that can be obtained using a single image. For instance, Sentinel-2 SITS have been successfully used to produce land cover maps at country scale and to characterize grassland area as proxy indicator for biodiversity, food production, and global carbon cycle. Combinations of SITS, e.g., optical (Landsat-8) and radar (Sentinel-1) SITS, have also been proposed in order to improve land cover maps on the agricultural domain.
Unfortunately, machine learning approaches commonly leveraged in remote sensing field fail to take advantage of spatio-temporal dependencies present in such data. Conversely, most of the strategies proposed for SITS analysis tasks directly apply standard techniques (i.e. Random Forest, SVM) on the stacked images, thus ignoring any temporal dependencies that may be discovered in the data. Indeed, such algorithms assume that the information (spectral bands and timestamps) are independent from each other.
Recently, deep learning methods allowed to significantly advance research in this field. These approaches have generally focused on a single type of neural network, i.e., Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), which model different but complementary information: spatial autocorrelation (CNNs) and temporal dependencies (RNNs).
Even though both approaches have been shown to be effective on the analysis of SITS data, our hypothesis is that, since they capture different knowledge aspects, a combination of both would produce a more diverse and complete representation of the information. For this reason, we propose a deep learning architecture for the analysis of Sentinel-2 SITS, namely DuPLO (DUal view Point deep Learning architecture for time series classificatiOn), based on the combination of Convolutional and Recurrent neural networks. To the best of our knowledge, DuPLO is the first example of deep learning architecture combining RNN and CNN approaches for the analysis of satellite image time series.
The figure shows a visual representation of the DuPLO Deep Learning architecture. The Sentinel-2 SITS in input is first processed independently by the two branches, consisting of two different neural networks, i.e., a CNN and a RNN. The result of each branch is a feature vector summarizing the extracted knowledge. The features extracted by each branch of the architecture are combined together, resulting in a single feature vector that is used to feed the final classifier that produces the land cover decision.
To further strengthen the complementarity as well as the discriminative power of the learned features for each branch, we add two auxiliary classifiers, linked to each group of features, in addition to the classifier that uses the concatenated information. The complementarity is enforced by alternating the optimization of the parameters of the two branches.
Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the Gard site in France and the Reunion Island in the Indian Ocean), demonstrated the effectiveness of our proposal compared to state of the art approaches for the characterization of land cover mapping on SITS data. The ablation analysis confirmed that the full DuPLO architecture always achieves better performance than its components taken separately, i.e., the CNN branch, the RNN branch, and the full architecture without auxiliary classifiers. A comparative analysis showed how DuPLO is able to outperform state of the art competitors (i.e., a standard Random Forest and a RNN based approach) as regards both quantitative and qualitative results on both study sites, emphasizing how the combination of CNN and RNN is beneficial for the classification task compared to the use of a single neural network model.

Area C - Board 327

Now there are a lot of studies that are investigating different methodologies for obtaining land cover and crop classification maps based on satellite data [1]. Such maps provide basic information for many other applied problems and are vital in remote sensing studies. Despite of which machine learning methods were utilized for maps obtaining and which source of imagery some noise could be still present on final maps, especially when pixel-based classifier used [1]. Traditionally in remote sensing community utilized different approaches and filters before classification phase in the stage of satellite data preprocessing for decreasing level of noise in crop classification map. But there is a lack of available good enough methods for specific area like crop classification maps postprocessing (removing noise on final map) [2]. At the same time deep learning is a powerful part of machine learning methods that have been applied to solve such problems as signal processing, speech recognition, social network filtering, web search, image processing, computer vision, translation from one language to another and remote sensing. Convolutional neural networks (CNN), deep autoencoders (DAE), recurrent neural network with Long Short-Term Memory (LSTM) model and deep belief networks (DBN) have been used for remote sensing (RS) problems [1, 3].
In this paper we propose method for land cover and land use classification maps filtration using a deep convolutional encoder-decoder network. Experiment in this study has been conducted for 10 m resolution crop classification map for Kyiv region (Ukraine) for 2018 obtained based on Sentinel-1 and Sentinel-2 imagery using pixel-based multi-layer perceptron. We selected several parts of crop classification map, which represent different land cover types and crop diversity, and manually remove all noise from them and refined fields boundaries. In such way reference data for training deep convolutional encoder-decoder network has been prepared. The accuracy and effectiveness of this method have been tested on the independent test set and using the visual comparison with the results of utilizing common filters. The detailed results will be presented within the conference.

References
1. N. Kussul, M. Lavreniuk, S. Skakun, and A. Shelestov, “Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data,” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 5, pp. 778-782, 2017.
2. M. Lavreniuk, N. Kussul, A. Shelestov, O. Dubovyk, and F. Löw, “Object-based postprocessing method for crop classification maps,” IEEE Geoscience and Remote Sensing Letters, 2018.
3. J. Geng, J. Fan, H. Wang, X. Ma, B. Li, and F. Chen, “High-Resolution SAR Image Classification via Deep Convolutional Autoencoders,” IEEE Geoscience and Remote Sensing Letters, vol. 12, no. 11, pp. 2351–2355, 2015.

Area C - Board 328

INTRODUCTION
The increased availability of a large amount of spatio-temporal images is driving the need for adequate tools to process, analyse and extract actionable information from the data. Machine learning (ML) and, in particular, deep learning methods have become the state-of-the-art in many vision, language, and signal processing tasks, due to their ability to extract patterns from complex high-dimensional input data. Classical ML methods, such as random forest and support vector machines [1, 2] have been used in many EO applications to analyse temporal series of remote sensing images. On the other hand, convolutional neural networks have been employed to analyse the spatial correlations between neighbouring observations, however mainly in single temporal scene applications [3]. In this paper, similarly to the work of [4], we investigate a deep learning architecture capable of simultaneously analysing the spatio-temporal relationships of satellite image series. In this work we demonstrate its application on the land cover classification of the Republic of Slovenia, using annual Sentinel-2 satellite images for the year 2017.

METHOD
We present a Temporal Fully-Convolutional Network (TFCN) that extends the FCN architecture (a.k.a. U-Net [5]), which is currently the state-of-the-art in single scene semantic segmentation. Similar to [4], the architecture exploits spatio-temporal correlations to maximise the classification score, with the additional benefit of representing spatial relationships at different scales due to the encoding-decoding U-Net structure.
The input tensor of the time series has a shape of [T, H, W, D], where T is the number of time frames, and H, W and D are the height, width and number of bands (i.e. depth) of the input image tensor. The algorithm performs a 3D convolution in the spatial as well as the temporal dimension. By default, max-pooling is performed in the spatial domain only. As the target land cover labels are not-time dependent (i.e. one label per pixel is available for the entire time-series), 1D convolutions along the temporal dimension are performed in the decoding path of the architecture to linearly combine and reduce the temporal features. The schematic of the model architecture is shown in Figure 1. The output of the network results in a 2D label map which is compared to the ground-truth labels. The architecture performed three encoding and decoding steps (i.e. three max-pooling and three deconvolution layers), with a bank of two convolution layers at each encoding and decoding scale. The number of convolutional features was set to 16 at the original scale, with a factor of 2 applied at each deeper level, with a kernel width of 3. The Adam optimiser (learning rate 0.001) was employed. The TFCN model is implemented in TensorFlow. The code to reproduce the entire pipeline will be released at the time of the conference.
FIGURE 1

EXPERIMENTS
To evaluate the TFCN, we present an end-to-end workflow to generate a land cover map for Slovenia. We do this by using eo-learn, a framework to handle multi-temporal and multi- source satellite data, both in raster and vector format. The framework allows splitting the area-of-interest (AOI) into smaller patches, that can be processed with limited computational resources and allows automation of the processing pipeline. The framework consists of open- source Python packages and is designed to facilitate the prototyping and building of Earth Observation applications. We use this framework to process the data and train the ML models to predict classes of land use and land cover.
The framework was used to generate a land cover map of the Republic of Slovenia for the year 2017. The inputs to the framework are a shape-file defining the geometry of the AOI, the Sentinel-2 L1C images for the entire year, and a set of training labels. Avoiding to download and process entire tile products (e.g. Sentinel-2 granules) provides flexibility and facilitates an automation of the processing pipelines. A pipeline in eo-learn is defined as a connected acyclic graph of well-specified tasks to be performed on the data. eo-learn supports parallelisation of operations, such that the same workflow (e.g. data preparation for land cover classification) can be run in parallel for the smaller patches constituting the AOI. Logging and reporting allow to monitor and debug the execution of the processing pipeline. The automated pipeline for predicting the land-cover labels follows the following setup:
1. The boundary of the AOI was split into 25 × 17 equal parts, which resulted in about 300 patches of about 103 × 103 px2 at a 10 m resolution. The splitting choice depends on the amount of available resources. The output of this step is a list of bounding boxes, covering the AOI.
2. The bounding boxes were used to create patches, or containers, where the information for the corresponding area is stored. Given the time interval of interest (e.g. from 2017-01- 01 to 2017-12-31), eo-learn used the sentienlhub-py package to download the desired band data. The s2-cloudless cloud detector was subsequently used to obtain the cloud probabilities and cloud masks. The following Sentinel-2 features were input to the model (bands B02, B03, B04, B08, B11, B12, their euclidean distance NORM, and indices NDVI and NDWI).
3. Ground truth information was obtained in a vector format, which was then rasterised over the given bounding box and added to the patch. The ground-truth is openly available through the Slovenian Ministry of Agriculture, Forestry and Food, and is mapped to the following classes: cultivated land, forest, grassland, shrubland, water, wetlands, artificial surface, and bareland.
4. A random spatial sampling of patches was performed to select time-series of pixels which are used for ML model training. The random sampling is uniform throughout the patch and independent on the labels.
5. Due to the non-constant acquisition dates of the satellites and the irregular weather conditions, missing data is common in EO. To deal with these, the mask of valid pixels in the time series is used to interpolate the values and fill the gaps due to the missing data. Linear interpolation was used, after which the time-series was sampled at uniform temporal dates (with a 16-day interval) to unify the dates among all the patches. This normalisation generalises the model to different AOI and time intervals.
6. The data was split into the train, cross-validation and test sets (60:20:20) and provided to the deep learning model, which was trained on an Amazon Web Services (AWS) instance.

RESULTS
The trained model was used to predict the labels on the test sample and the obtained results were then validated against the ground-truth. An overall accuracy of 84.4% and a weighted F1 score of 85.4% were achieved. In general, poor prediction was achieved for under-represented classes such as wetlands and shrubland. These results represent preliminary work on a prototype architecture which was not optimised for the task at hand. Despite this, results in line with previously reported work (e.g. Inglada et al. [1]) were achieved. Optimisation of the architecture (e.g. number of features, depth of the network, number of convolutions) and of the hyper-parameters (e.g. learning rate, number of epochs, class weighting) is required to fully assess the potential of TFCN.

DISCUSSION AND CONCLUSIONS
In this paper we have presented an approach to a land-use-land-cover classification using deep learning. While the architecture of the presented TFCN model was not fully optimised, it shows promising results, with the overall accuracy and weighted F1 score reaching levels of about 85%. A thorough and extensive evaluation comparing the TFCN with state-of-the-art methods will be presented at the symposium. Insights into good practices for training, evaluating, and deploying machine learning and deep learning models for EO applications will also be discussed. Source code to reproduce the framework and the models will be made available at the time of presentation to foster research and development in the exploitation of spatio-temporal data in EO applications.

REFERENCES
[1] Inglada J., Vincent A., Arias M., Tardy B., Morin D., and Rodes I., “Operational High-Resolution Land Cover Map Production at the Country Scale Using Satellite Image Time Series“, Remote Sensing 9(12), DOI: 10.3390/rs9010095, 2017.
[2] Karakizi C., Karantzalos K., Vakalopoulou M., Antoniou G., “Detailed Land Cover Mapping from Multitemporal Landsat-8 Data of Different Cloud Cover“, Remote Sensing 10(8), DOI: 10.3390/rs10081214, 2018.
[3] Helber P., Bischke B., Dengel A., and Borth D., “EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification“, arxiv: 1709.00029, 2017.
[4] Russwurm M., and Koerner, M.,“Multi-Temporal Land Cover Classification with Sequential Recur- rent Encoders“, ISPRS International Journal of Geo-Information 7(4),DOI: 10.3390/ijgi7040129, 2018.
[5] Long J., Shelhamer E., and Darrell T., “Fully convolutional networks for semantic seg- mentation“, 2015 IEEE Conference on Computer Vision and Pattern Recognition, DOI: 10.1109/CVPR.2015.7298965, 2015.

Area C - Board 329

Water lies at the heart of economic and social development. As it is becoming scarce, stakeholders will need innovative ways to better understand water conditions, predict risks, and tackle problems. Cost-effective, yet reliable solutions for monitoring water resources are needed, as ground-based monitoring networks are often too costly and due to networks deterioration in some cases also unreliable. Developing countries in particular need cost-effective, yet reliable solutions and tools for monitoring available water resources.

Being enlightened by JRC’s Global Surface Water project [1] we have set to build a service, which does not only show historic data but is also up-to-date. Copernicus Sentinel mission, with its global coverage and short revisit time, combined with an efficient use of cloud infrastructure resources makes it feasible to do a global scale project with limited resources. The Blue Dot Water Observatory[2] is a showcase of this approach, both as an operational service as well as a set of open-source tools, which can be modified to fit specific needs.

The Blue Dot Water Observatory is an Earth-observation-based solution that provides reliable and timely information about surface water levels of water bodies across the globe. All observations are provided and can be explored interactively via the Water Observatory Dashboard [3] or via RESTful API. The key benefit of the service is the accumulation of current and historic surface water level data in one place, presented in a clear and interactive way, free of charge. The Water Observatory provides a valuable service to local authorities, governmental agencies, natural parks and reserves, agricultural ministries and agencies, stakeholders in food and energy production, and citizens alike.

With this service, we also wish to demonstrate how global monitoring of the environment using Earth observation data can be done efficiently and orders of magnitude cheaper than before, if done in an intelligent way. Benefiting from lessons learned from similar past projects, we will share as an open-source the code, water detection algorithms as well as details on how to put service like this in production. We hope this will inspire others to build on top of it and develop similar services for other use cases.

In addition, we plan to share our database of water bodies that we have built on top of exiting ones, like The Global Reservoir and Dam Database (GRanD), and WWF’s Global Lakes and Wetlands Database. The resulting historic trends of surface water levels provided by the service can be also serve as an important input to research related to climate change, impact of climate change to food and energy production, and similar.

[screenshot of the dashboard to be added]


[1] Jean-Francois Pekel, Andrew Cottam, Noel Gorelick, Alan S. Belward, High-resolution mapping of global surface water and its long-term changes. Nature 540, 418-422 (2016). (doi:10.1038/nature20584)
[2] https://www.blue-dot-observatory.com
[3] https://water.blue-dot-observatory.com

Area C - Board 330

The deep learning revolution in computer vision has enabled a potential for creating new value chains for Earth observation (EO) that significantly enhances the analysis of satellite data for tasks like land cover area mapping, change analysis, and object detection. In particular, for very high-resolution EO data, deep learning based methods has demonstrated results that were not achievable 5 years ago.

A key innovative aspect of deep learning based value chains is that they, through learning from data, are flexible, generic and easily adaptable to new applications. Deep learning based value chains are often realized using deep convolutional neural networks. For such value chains, the tasks define the data and type of annotation needed, and deep networks to use in order to solve the tasks. The data are then used to learn the network parameters using a learning framework.

A key characteristic of deep learning based value chains is that the network architectures are mainly dependent on the tasks, and to a lesser degree the data. The dependency between the data and network is encoded in the parameters being learned. Hence, the same network may be used to perform agriculture mapping using Landsat data as forest mapping using Sentinel-2 data, or urban land cover mapping using aerial images. However, there exist situations where the network may depend on the data. For instance, we may use a different network if we have multi-modal data compared to single-modal data, or if we have labels in form of semantic descriptions compared to complete pixel labels.

We demonstrate a deep learning based value chain for the task of mapping forest in the Liwale region in Tanzania using Sentinel-2 data. As ground truth data we use lidar measurements, which are process to provide the average tree height per Sentinel-2 pixel (10m). The network we apply is the so-called UNet, which is a widely used network for segmentation tasks in computer vision. However, other networks may also be used. We demonstrate the value chain be for both regression and classification tasks. In the classification task, the aim is to perform a forest extent mapping by classifying pixels in to less than 2m and above 5m average vegetation height. In the regression task, the aim is to estimate the average tree height.

Preliminary results show that we are able to map the forest extent with high accuracy. For the task of estimating tree heights, we observe that the tree height predictions saturates for values above 10m. The preliminary results also show that the performance depends on the greenness of the vegetation. Higher performance scores are obtained when the vegetation is green (high NDVI values) compared to cases when the vegetation is dry (low NDVI values).

Area C - Board 332

In the last decade we witnessed a massive technological explosion in Computer Science, particularly the Cloud gained increased attention both inside and outside of the IT community. Also in the last few years there was a massive shift towards Machine Learning (ML), with many IT companies investing significant resources in developing new techniques and products aimed at bringing ML closer to the developer communities, and finally to the end-user. All this evolutions in the IT and computer vision community resulted in advanced technological solutions which unfortunately did not get the expected adoption rate in the Earth Observation (EO) community. There is reticence in adopting new ML techniques, partially motivated by the complexity of deploying and setting up the technologies. Two major trends have arisen in recent years. First the proliferation of Deep Learning techniques and second the availability of up to date high quality satellite imagery such as those from Sentinel-2 from the Copernicus program. As a result research into using deep learning techniques and other ML techniques for satellite imagery is on the rise. This is mainly due to the fact that once a deep network is trained its computational tractability is far superior to any of the existing methods. The major downside to this is that training it is tricky requiring a well thought out training and validation processing pipeline.

The overall goal of the ML4EO (Machine Learning for Earth Observation) project is to address the lack of adoption of Cloud and ML technologies by proposing a software stack aimed at simplifying the access to this technologies. As with many technologies the adoption of novel ML methods and tools brings with it a steep learning curve for the end-user. This learning curve may hinder the adoption of such solutions. Specialized ML platforms like Theano, Caffe, TensorFlow have the potential to perform well in use cases linked with EO. However, most end-users interested in processing this kind of imagery might not have the technical know how or time to learn how to use them. We propose to facilitate the proliferation of novel ML frameworks in the area of EO by creating a solution which abstracts as much as possible the underlying technological stack, exposing only what is strictly necessary for the EO expert user.
Some of the generic technical objectives of the ML4EO project are to:
- Investigate and create a novel technology, based on the latest ML developments, for the integration and processing of large EO and non-EO datasets
- Facilitate the development/enhancement of the classical data processing workflows by making available big data tools and models as services and exploiting new paradigms in EO;
- Showcase for testing and evaluating the viability of the ML techniques implemented in ML4EO for operational use;
- Contribute to a better understanding of some Romanian specific environmental problems by using new approaches in EO processing and by involving new national and regional stakeholders;
- Create robust opportunities for Romanian entities to remain at the forefront of science, technology and innovation in the exploitation of EO data;
- Develop a technology that will ready to be used in conjunction with the future Romanian Copernicus Collaborative Ground Segment.

The developed platform will be extensively tested/validated using EO data for Romania (especially Sentinel-1 and Sentinel-2), along with reference and ground truth data from the most relevant data producers and custodians. These include but are not limited to: (1) High resolution (0.5m cell size) orthophotos from 2005-2018), (2) SPOT 2.5 m cell size high resolution images 2008 to 2016, (3) Up to date anonymized Land Parcel Identification System database, (3) Permanent water bodies for Danube and Danube Delta, (4) curated forest clear cuts masks etc.
The training datasets, along with the software modules of the platform will be published under appropriate open licences.

Area C - Board 333

In the past years, the quality and quantity of Earth observation imagery has been steadily increasing. The Sentinel-2 constellation provides time series with a short revisit of 5 days, at a spatial resolution of 10m, which allows for the precise monitoring of natural, agricultural, and artifical surfaces, at a near real-time pace. Very high spatial resolution instruments like SPOT7 provide sub-metric optical images which contain an unprecedented level of detail, like textures, and the precise contour of image objects. This affluence of new data has been challenging the remote sensing community to enhance the precision and class variety of yearly land cover maps. In particular, certain classes that are characterized by spatial arrangement of pixels in a wider area should be discriminable using high or very high spatial resolution images. Several studies have shown that the inclusion of context in the classification can greatly improve the discrimination of many land cover classes, for example, for distinguishing different levels of urban density, orchards, or vineyards. In this study, three types of supervised contextual classifiers will be compared on their ability to recognize these sensitive classes. A popular approach nowadays is to train a Deep Convolutional Neural Network to extract the optimal contextual features, in a supervised, end-to-end framework. Here, two main types of architecture can be distinguished. The first consists in using convolutional and pooling layers, followed by a fully connected layer to provide one label for each image patch, which is then assigned to the central pixel of the patch. The other method involves using convolution filters followed by deconvolution filters, in order to assign a dense labeling to all of the pixels in the patch. This architecture is the base of the U-Net architecture, and is usually faster than the patch-based networks, as it has fewer variables to optimize. A more detailed description is given in [1]. However, applying such methods to data with a very large dimension, like multi-spectral time series of images, can be difficult due to the large number of variables to optimize. Furthermore, in the absence of densely labeled data, these methods are prone to oversmooth the output maps, particularly in high spatial frequency areas, like sharp corners or fine elements (roads, rivers). At the opposite of these approaches which use an end-to-end optimization, there are methods that involve manually extracting the contextual features in combination with a supervised classifier like Random Forest or Support Vector Machine. This requires first selecting a neighboring environment, which can be either a fixed shape (square, circle), or a segment from an image segmentation, as in Object Based Image Analysis [2]. Then, certain features are calculated within the neighborhood, like textures, gradients, etc. These are then used by the classifier to characterize context dependent classes. The choice of neighborhood size, shape, and the nature and quantity of features is not simple, and relies on strong human expertise, but these methods are usually faster. In this study, superpixel segments [3] are investigated as a spatial support, as they provide adaptive neighborhoods with a controllable size. In terms of features, the result of a dense pixel-based classification is used to calculate class histograms within these neighborhoods, in a way similar to the Auto-Context method. The objective of this study is to compare the performance of three types of contextual classifiers, one of each of the two Deep Learning architectures described above, and one involving hand-crafted features calculated in superpixel segments, on two data sets. The first is a 14 class problem, using high spatial resolution Sentinel-2 multi-spectral time series. This data set contains 11 different 110x110km areas across the French territory, in order to cover a variety of landscapes and eco-climatic areas, and is based on a real-world land cover mapping application [4]. The second data set contains 5-classes, and is based on very high spatial resolution SPOT7 optical data. These three methods are compared on their ability to enhance the precision of land cover classification, but also on their computational cost, and on the geometric quality of the maps, in other words, the amount of smoothing in high spatial resolution areas. This in-depth comparison allows for a better understanding of the advantages and drawbacks of Deep Learning methods, when applied to various land cover mapping problems. Our experiments show that shallow learning with the appropriate hand crafted features is competitive with the studied Deep Learning approaches in terms of accuracy, while having a much lower computational complexity. This talk will present a detailed analysis of the assessment in terms of classes and types of landscape, giving insights for the choice of the appropriate methodology in operational settings.

[1] Maggiori, E., Tarabalka, Y., Charpiat, G., & Alliez, P. (2017). Convolutional neural networks for large-scale remote-sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(2), 645-657.

[2] Blaschke, T. (2010). Object based image analysis for remote sensing. ISPRS journal of photogrammetry and remote sensing, 65(1), 2-16.

[3] Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunk, S. (2012). SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11), 2274-2282.

[4] Inglada, J., Vincent, A., Arias, M., Tardy, B., Morin, D., & Rodes, I. (2017). Operational high resolution land cover map production at the country scale using satellite image time series. Remote Sensing, 9(1), 95.

Area C - Board 334

In this study, we address the problem of change detection using Sentinel-2 data. The continued development of infrastructures such as buildings and roads has increased the attention toward the soil consumption. The latter occurs often in low density urban areas, at European level as well as in developing countries, which are subject to rapid urbanization processes. A detailed knowledge of such changes across time is highly demanded since it can be useful to understand the urbanization process and make proper urban planning management.
The launch of the Copernicus Sentinel-2 satellites allows us to address several remote sensing problems by use of multi-spectral data with a maximum spatial resolution of 10 m, 13 bands, wide acquisition coverage and short time revisit. The availability of the Sentinel-2 huge amount of open data and the recent development of novel machine learning techniques allow to improve the accuracy of change detection applications.
In this work we seek to address the problem of identifying artificial changes altering the natural landscapes, using the Sentinel-2 products and Convolutional Neural Networks (CNNs). The latter have become a state-of-art solution for several computer vision problems, such as image classification, object detection, semantic segmentation, etc. CNNs can learn hierarchical image representations from the input data by successively abstracting higher-level features from the previous low-level ones. In the last years, this type of neural networks has been used to solve several tasks in different domains. In particular, CNNs have demonstrated excellent performances in different remote sensing problems, such as pixel-based classification, target re- cognition and image pan-sharpening.
CNNs have also shown great capability in the extraction of image’s features, the utiliza-tion of which for processing remotely sensed imagery is relatively recent.
The proposed method is based on a Fully Convolutional Neural Network with a siamese architecture optimized through a local similarity loss function. Siamese neural network [1] is a class of neural network architectures that contain two or more identical subnetworks. More specifically, the term siamese means that the subnetworks have the same configuration with the same parameters and weights. The parameter updating is mirrored across both subnetworks. Each subnetwork takes one of the two input images, which here represent two Sentinel-2 products, acquired at different time, considered to extract the artificial changes over the images.
In general, in change detection applications the images acquired at different time are concatenated at the input layer of the net. The proposed approach, using the siamese network in the encoder part as fusion strategy, allows us to combine the features extracted from the two input images in a more suitable way. Indeed, the capability of the convolutional layers to extract images features are exploited for the two input images separately.
The study of Daudt et al. [2], has already shown, with a simple Unet architecture [3], that this kind of approach performs better than a typically fully convolutional deep network, since the siamese strategy fusion guarantees a better optimization. To further improve the results, it would be necessary to create a more deeper neural networks to extract more information from the data. Anyway, it is empirical demonstrated [4], how at the network depth increasing, the accuracy gets saturated and then degrades rapidly. In the present work we studied a solution for this issue demonstrating how the use of the recent ResNet architecture [4] in a siamese model ensures a better convergence capability during the training phase.
The change detection algorithm is developed considering that, in a siamese architecture, each subnetwork essentially produces a representation of its input. If the inputs represent the same context, it makes sense to use similar model to process similar inputs. Therefore, we exploited a local similarity loss function to force the siamese encoder to learn how to map, into a lower dimensional space, each local region of a Sentinel-2 image. If no changes occurred, the siamese encoder extracts similar features, otherwise the features representations of the inputs are located distant in the dimensional space and are consequently classified as changes.
To evaluate the proposed method, we used the Onera Satellite Change Detection dataset [5] (OSCD). The experiments demonstrate the effectiveness of the proposed technique with respect to other consolidated methods.

[1] Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a Similarity Metric Discriminatively, with Application to Face Verification. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01 (CVPR '05), Vol. 1.

[2] Caye Daudt, R.; Le Saux B & Boulch, A. 2018. Fully Convolutional Siamese Networks for Change Detection, IEEE International Conference on Image Processing (ICIP). Athens, Greece.

[3] Olaf Ronneberger, Philipp Fischer & Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation.Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, LNCS, Vol.9351, 234--241, 2015

[4] He, Kaiming, Xiangyu Zhang, Shaoqing Ren and Jian Sun. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 770-778.

[5] Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks, R. Caye Daudt, B. Le Saux, A. Boulch, Y. Gousseau. IEEE International Geoscience and Remote Sensing Symposium (IGARSS’2018). Valencia, Spain. July 2018.

Area C - Board 335

Introduction

Change detection (CD) has been one of the main topics in remote sensing for many decades, and is still a very active field with large potential. The goal of CD in earth observation is to identify areas which have changed, by the comparison of two or more satellite images acquired at different times. Applications range from damage assessment after natural disasters, such as floods, earthquakes, and avalanches, to the monitoring of long-term trends in land usage, urban development, glacier dynamics, deforestation, and desertification [1 - 2].

Traditional CD methods are based on the assumption that the images are homogeneous, i.e. acquired with the same sensors, same modalities, same geometries, and same environmental conditions. However, for many practical examples and applications this assumption does not hold true. Moreover, the development and deployment of new and different satellites and on-board sensors increased the quantity and typology of data, which unfortunately cannot be entirely exploited by homogeneous CD techniques [1 - 2].

These reasons have brought the remote sensing community to focus on a more challenging task, the so-called heterogeneous CD. Data coming from heterogeneous sources provide more flexibility, both to assess the aftermath of sudden catastrophic events promptly and to monitor long-term trends with a higher temporal resolution. However, the biggest problem with multimodal data is that they are defined in different and unrelated domains, and a direct comparison yields no meaningful results. To overcome this issue, a highly nonlinear transformation of the data is often necessary to adapt the data to the same domain.

Method

In this work we propose an unsupervised framework for bi-temporal heterogeneous change detection based on deep learning. In particular, we use convolutional autoencoders (AEs) to learn new representations of the data and generative adversarial networks (GANs) to align the distributions of these representation. The goal is to bring the data coming from two different domains into a common one, where they can be compared by means of the traditional methods of homogeneous CD.

Two distinct AEs, one for each of the image modalities, are trained separately and independently from one another. The AEs capture spatial information from the images and obtain a new meaningful representation of them in a code space. However, this does not imply that the two code spaces are necessarily the same and that the underlying distributions of the code layer data are aligned.

To remedy this we propose the use of GANs. Indeed, they are particularly suitable for the task: a generator, which tries to emulate data coming from a source domain, is counterposed to a discriminator, a classifier which instead is required to distinguish fake data from real data. In particular, domain-adversarial training consists in teaching two different encoders to transform the data to a code space where they are not distinguishable, maximising the discriminator loss, also called adversarial loss or domain confusion loss. Furthermore, other examples and implementations contemplate even more contributions in the domain confusion loss. For example, one could generate fake data from the code representation 1 (2) by using the decoder of the AE 2 (1). If a discriminator cannot distinguish such fake data from the original input data, it means that the codes produced from both AEs are similar enough to be reconstructed in both input domains. The attached figure illustrates the concept.
This kind of methodologies has already proved successful in the field of unsupervised domain adaptation in deep learning [3 - 7].

Concerning our proposed method, instead of training independently the two AEs, we require not only to reconstruct the original data, but also to yield code representations that are as similarly distributed to each other as possible.The adversarial fashion of this training phase let us achieve the goal of pulling the two code spaces together, so that the data can be directly compared. Ideally, if the distributions of data in these spaces are perfectly aligned, the decoder of the first (second) AE can be used to decode the code representation of the image at the second (first) acquisition time to transfer it into the input space of the image at the first (second) time.

It is important to underline that the whole process is unsupervised, since the AEs are trained to reconstruct the input, without any need of labels. Training the discriminator does not require supervision either, since it only needs to distinguish whether the provided examples of code come from the first or the second AE, without any knowledge about the data itself. This property marks an important difference with respect to previous heterogeneous CD methods published so far, most of which explicitly make use of prior information through supervised, semi-supervised, or semi-interactive approaches.
Finally, the authors' intention is to extend this methodology by introducing the use of affinity matrices, either in the input space or in the code space, to include structural information of the data as a key element of the training phase. However, it is still premature to formally define how this can take part of the methodology so further investigation is needed.

Data sets

The methodology is tested on two data sets. The first data set consists of two optical images sensed before and after a forest fire in Bastrop County, Texas, during September-October, 2011. The Multispectral Scanner sensor mounted on Landsat 5 acquired the first image; the Advanced Land Imager equipped on Earth Observing-1 acquired the second image. The second data set involves an optical image (Operational Land Imager, Landsat 8) and a synthetic aperture radar image (Sentinel-1A) covering Sutter County, California, before and after a flood in January 2017.

References

[1] Liu, Z., Li, G., Mercier, G., He, Y., & Pan, Q. (2018). Change detection in heterogenous remote sensing images via homogeneous pixel transformation. IEEE Transactions on Image Processing, 27(4), 1822-1834.

[2] Luppino, L. T., Anfinsen, S. N., Moser, G., Jenssen, R., Bianchi, F. M., Serpico, S., & Mercier, G. (2017, June). A clustering approach to heterogeneous change detection. In Scandinavian Conference on Image Analysis (pp. 181-192). Springer, Cham.

[3] Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015). Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision (pp. 4068-4076).

[4] Murez, Z., Kolouri, S., Kriegman, D., Ramamoorthi, R., & Kim, K. (2017). Image to image translation for domain adaptation. arXiv preprint arXiv:1712.00479, 13.

[5] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1), 2096-2030.

[6] Luo, Z., Zou, Y., Hoffman, J., & Fei-Fei, L. F. (2017). Label efficient learning of transferable representations acrosss domains and tasks. In Advances in Neural Information Processing Systems (pp. 165-177).

[7] Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015). Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision (pp. 4068-4076).

Area C - Board 336

This work intends to explore the potential of radar data for burnt area mapping, as an alternative to optical mapping techniques. Due to the fact that optical data acquisition is often limited by cloud cover, while radar occupies all-weather and day-and-night sensing ability, therefore, radar-based mapping technique will play an increasingly important role in monitoring wildfire progression[1,2].
In recent years, deep learning-based models have won numerous contests in pattern recognition and machine learning, and the success lays in its layer-wise and powerful representation learning ability.
Convolutional Neural Networks (CNN) is one of the most powerful deep learning models[3], in this research, CNN's powerful nonlinear learning ability is exploited for burnt area mapping based on Sentinel-1 C-band SAR data. Our study event is Thomas wildfire, occurred in Southern California, USA, which was a massive wildfire that affected Ventura and Santa Barbara counties. It burned approximately 114,078 hectares, becoming the 2nd largest wildfire in modern California history.

The proposed CNN-based burnt area mapping framework is shown in Fig.~(a), which is based on pre-fire and post-fire SAR data. First, we derive a fused change map using interferometric SAR (InSAR), based on which, an initial segmentation map can be generated for providing Pseudo labels with Chan-Vese method. Additionally, a superpixel-based sampling strategy is adopted to select reliable training samples for CNN. For each superpixel, a local neighborhood is extracted centering at the geometric center of superixel, and meanwhile, the corresponding Pseudo labels are obtained with majority voting. Having the input and the corresponding labels, a CNN model can be trained with back-propagation algorithm. In our experiment, the CNN model is trained on two pre-fire dates (20171128-DSC144 and 20171204-ASC137) and two on-going dates (20171210-DSC144 and 20171216-ASC137), and it is transferred on two other unseen dates (20171222-DSC144 and 20171228-ASC137). Based on different polarization modes, the corresponding burn mapping results are listed in (b)-(g). The comparison between VV-based and VH-based change maps shows that VH is more sensitive to the structural changes caused by wildfire events. It is still hard to highlight the burnt area with VV only even by learning an enough complex transformation with CNN. The burn probability maps predicted by CNN indicate that VH works much better than VV in highlighting the burnt area, while VV+VH can achieve a slightly better results than VH only. Compared with log-ratio, CNN can be trained to find a better difference operator for pre-fire and post-fire C-band SAR data.
This study demonstrates the potential of radar data for burnt area mapping, and CNN is proven promising in highlighting and detecting the burnt area with Sentinel-1 data.

Reference
1. Tanase, M. A. et al. Sensitivity of X-, C-, and L-band SAR backscatter to burn severity in mediterranean pine forests. IEEE Transactions on Geosci. Remote. Sens. 48, 3663–3675 (2010).
2. Imperatore, P. et al. Effect of the vegetation fire on backscattering: an investigation based on Sentinel-1 observations. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. 10, 4478–4492 (2017).
3. Zhu, X. X. et al. Deep learning in remote sensing: a comprehensive review and list of resources. IEEE Geosci. Remote. Sens. Mag. 5, 8–36 (2017).

Area C - Board 337

Satellite data from the space missions of the European Copernicus program offers an operational data source for forest monitoring. In particular, the Sentinel-2 mission that offers optical imagery at 10-meter spatial resolution more frequently than once a week free of charge can change the paradigm in forest inventory. The methods for data utilization have to be able to absorb and process huge masses of data. Automatic processing of massive data volumes offers an opportunity to retrieve new and improved information on forest resources. Increased automation and improved accuracy in the remote sensing service production is expected to lead to an impact of increased efficiency in the operations of the clients.

We will develop a concept to apply deep learning and related methods for the prediction of forest variables from Sentinel-2 satellite data. Results of the development will be demonstrated in the Forestry Thematic Exploitation Platform (F-TEP) platform. The application of deep learning methods to satellite images is not straightforward. The image frames contain typically gigabytes of data and feeding them into deep networks as such is not feasible. Also tailored solutions for image data organization (including pre-processing) will be needed. Typically image data is fed to deep learning network as image patches or sub-windows. One key issue to examine is the size of these image patches to optimize the predictive performance and on the other hand to minimize the computational burden.

The most notable applications of deep neural networks (DNNs) in remote sensing include land cover classification with optical images (Chen et al. 2014a, Romero et al. 2016) (one third of articles), hyperspectral image analysis (one in five articles) (Chen et al. 2014b, Tuia et al. 2015) or Synthetic Aperture Radar (SAR) image analysis (Geng et al. 2015) (one in eight articles). Change detection methods based on deep learning networks are also emerging (Lyu et al. 2016), with more than 12 articles. A large majority of published works use DNNs trained on patches extracted from satellite images. DNNs are usually not trained on databases of full sized satellite images (1 to several Gb per image) due to memory limitations, even on powerful GPU servers. Convolutional Neural Networks (CNN) are the most commonly used deep learning architectures for the classification of optical (Molinier et al. 2017) and SAR (Geng et al. 2015) satellite images. Recurrent Neural Networks (RNN) is another kind of deep network architecture that is suited for multi-temporal signals, and has been applied in satellite image change detection (Lyu et al. 2016).
Since large datasets of satellite images with high quality labels are not easily available, many works utilized pre-trained DNNs such as AlexNet (Krizhevsky et al. 2012) trained on computer vision benchmark datasets (ImageNet), not on satellite images (Marmanis et al. 2016, Anwer et al. 2017). Application of pre-trained networks to remote sensing images can be done through transfer learning (Hu et al. 2015) or domain adaptation (Penatti et al. 2015). A few recent works train DNNs on databases of actual satellite or aerial images, such as the SAT-4 and SAT-6 datasets (Basu et al. 2015) (respectively 500 000 and 405 000 patches of aerial images with labels for 4 and 6 classes), or using approximate labels (Mnih et al. 2010).

This study will focus on investigating the possibilities that the deep learning methodology will offer for the prediction of forest variables from satellite data, with the goal of reaching accuracy levels that are adequate for operative use in forest industry in their decision-making. Suitable methodology will be selected and applied in test area in Finland. Open forest variable data from Finnish forest centre will be used as reference data. The results will be compared with traditional methods to estimate forest variables from satellite data.

Area C - Board 338

Forests play a key role in climate change by regulating the CO2 content in the atmosphere. Not only do they absorb CO2 but they also store carbon in their trunks, branches, and roots. To assess the actual impact of forests on our climate, global biomass maps are essential. Today, the mapping of above ground biomass stored in forests relies mainly on vegetation height as it is the most important indicator.

Traditionally, vegetation height is mapped by measuring the difference between ground and top of canopy elevation. A straightforward strategy is the application of active LiDAR systems that can directly measure forest structures including the ground elevation by penetrating the canopy. Alternatively, photogrammetric stereo-matching yields the top of canopy elevation, but relies on highly overlapping multi-view aerial imagery. The main drawback of these approaches is that they rely on expensive flight campaigns. Therefore, they lack scalability and vegetation height products are rarely up-to-date.

This work addresses vegetation height mapping at country scale at 10 m spatial resolution by exploiting the recent success of deep learning methods combined with unprecedented publicly available multispectral satellite images from the Sentinel-2 mission. Multispectral satellite images such as provided by Landsat and Sentinel-2 have been widely used for land cover classification. Their spectral bandwidths, especially the near-infrared bands, are well suited to assess vegetation vitality and to classify species. However, modelling vegetation height from a single-view multispectral image is challenging because various land cover types and also vegetation heights can yield the same spectral signature. Hansen et al. (2016) show that incorporating time-series spectral information is valuable for discriminating tree heights. However, their proposed method cannot fully exploit spatial patterns correlating with vegetation heights.

Our proposed method builds on state-of-the-art Convolutional Neural Networks (CNNs) to solve this pixel wise regression task and model vegetation height from a single multispectral Sentinel-2 image by learning the spectral and spatial features in a supervised fashion, following the end-to-end deep learning paradigm.
Our method is experimentally evaluated on two regions in Switzerland (1,320,000 ha and 1,420,000 ha in size) for which a ground truth vegetation height model (VHM) from 2016 exists at a 1 m resolution. This ground truth VHM was produced at 1 m resolution by the Swiss Federal Institute for Forest, Snow and Landscape using photogrammetric stereo-matching from aerial images taken by swisstopo in 2016 to create the digital surface model (DSM). This DSM was reduced to the final VHM by subtracting a LiDAR based terrain model and masking the buildings. We projected and resampled the original 1x1 m VHM to match the 10 m ground sampling distance of Sentinel-2 images.

Each region is split into non-overlapping train and test regions. We take cloud free Sentinel-2 images from the growing season as input to estimate vegetation height at 10 m spatial resolution achieving an overall mean absolute error (MAE) of 1.9 meter. Evaluating MAEs for separate height intervals yields 1.0 m (0-10 m), 5.4 m (10-20 m), 4.8 m (20-30 m), 4.9 m (30-40 m), 19.6 m (40-50 m), and 47.6 (>50 m). At the moment, our model is not capable to predict heights >40 m. Because vegetation heights >40 m account for only 0.06 % of the ground truth data, the model is not able to learn the characteristics of this underpopulated range. Furthermore, our model tends to slightly smooth the predictions with respect to the ground truth (see Figure). We are currently working on the integration of time-series data as proposed by Hansen et al. (2016) to further improve our results and investigate the saturation problem for heights >40 m.
Nevertheless, our results show that estimating vegetation height from multispectral Sentinel-2 images is promising and possible at 10 m spatial resolution given a vast amount of training data. Our proposed method allows vegetation height mapping at country scale with the same high temporal resolution of (cloud free) Sentinel-2 images (approx. every 5 days) at virtually no costs.

References:
Hansen, Matthew C., et al. "Mapping tree height distributions in Sub-Saharan Africa using Landsat 7 and 8 data." Remote Sensing of Environment 185 (2016): 221-232.

Fig. 1:  Vegetation height prediction for a test area on the east side of Lake Zurich

Area C - Board 339

Due to global warming, natural disasters such as thunderstorms are becoming more frequent (Brooks, 2013). Such events have a strong impact on forest health, wildlife habitat and can inflict substantial economic losses. Methods to quickly assess damage and to manage disaster in the aftermath of storm events are of high importance for forest management.
As there are large areas affected by storms, assessing the forest damage is time consuming, if done manually using aerial imagery and GIS tools. Time is a crucial factor in disaster management and therefore integrating deep learning with remote sensing data into a GIS environment holds a high potential for accelerating and improving damage assessment.
Convolutional Neural Networks (CNN), a supervised learning architecture, is extensively used in many areas (e.g. interpretation of medical images, 3D reconstructions, objects detection and classification in self-driving cars…). They are efficient in finding patterns within a large amount of data and are widely used in computer vision and images classification (LeCun et al., 1989). We develop an algorithm based on CNNs using a U-net architecture that is trained on labeled damaged areas visible in after-storm aerial orthophotos of a ca. 109 square km forest area in Bavaria (RGB and NIR, 0.2 m spatial resolution). Integration into ArcGIS was achieved with the Python API for ArcGIS and Jupyter Notebooks.
The neural network was trained using over 10e7 pixels of labeled data. Prediction on an independent dataset was achieved within seconds (depending on the size of the input raster) and resulted in an overall classification accuracy of 92% and an intersection over union score of 0.67. These results are promising, considering the complexity of detecting areas of fallen trees. In addition, we found labelling errors in the ground truth that somewhat bias the results because the algorithm in some instance performed better than the human labelling procedure.
The next step will be to use a second CNN architecture (Mask-r-CNN) to quantify timber losses due to the storm in terms of volume in addition to areal damage. We will also consider further optimization of the algorithm by assessing the optimal ratio of true positives to false negatives, based on a decision analysis. Overall, our results highlight the potential of deep learning on high resolution imagery for damage assessment following disasters.

Y. LeCun , B. Boser , J. S. Denker , D. Henderson , R. E. Howard , W. Hubbard , and L. D. Jackel (1989): Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1:4, 541-551
Brooks, H.E. (2013): Severe thunderstorms and climate change. Atmospheric Research, pp. 129-138.

Area C - Board 340

Nowadays, thanks to the modern satellite constellations delivering high resolution optical data, we are able to study in deeper details many aspects of the Earth status. Also, due to the large amount of acquired data coming from several missions we can monitor constantly the Earth environmental condition.

Unfortunately, due to the physical nature of optical sensors, the acquired information suffers from several atmospheric natural events such as rain, clouds, fog, fire smoke and volcanic gases. All these phenomena, in particular the clouds, affect the availability of useful data for scientific purpose. Indeed, on average at any moment in the day, about 52% of Earth is usually covered by clouds. Some regions are almost always cloudy such as the Amazon Rainforest whilst some others are almost always clear such as the Sahara Desert. In addition to natural phenomena, we have to consider the revisit cycle of satellite too (for instance each of the Sentinel-2 satellite has an individual revisit cycle of 10 days at the equator with one satellite, and 5 days with 2 satellites under cloud-free conditions which results in 2-3 days at mid-latitudes) and that optical acquisitions are limited to daytime operations. These factors together reduce the amount of data available for any kind of application especially time series analysis over the same area. For example, in Northern Europe, the cloud free optical imagery decreases dramatically with respect to the total data acquired, affecting analysis based on optical time series.

Differently from optical data, that covers the range of 390 / 700 nanometres of the visible spectrum, SAR (Synthetic Aperture Radar) sensors exploits microwave wavelengths that are able to penetrate the clouds and so provide cloud-free products. These products, being the result of a reflected signal travelling not in the visible spectrum, cannot be rendered as natural colours images and usually are rendered as grey scale images or processed in order to produce an RGB image with fake colours highlighting specific features. This kind of colorization do not help visual interpretation in the same easily way as per optical counterpart.

Nowadays, in the Artificial Intelligence field, several studies, papers and applications are oriented in image and video restoration process (e.g. aged ancient photos or movies coming from black and white era) exploiting the possibility to inject missing colour inside monochrome images. Starting from this concept and considering a less variance in terms of possible different situation w.r.t to real-life scenes, the aim of the project is to provide a system, based on a Deep Neural Network, able to infer colour information inside SAR images to obtain synthetically reproduced optical data in natural colours totally cloud-free. The model is designed on a Convolutional Neural Network (CNN) trained with ad-hoc selected Sentinel-1 and Sentinel-2 (cloud free) products constituting our training and test data sets. Each single training example is based on a precise pair of Sentinel-1 (our input feature) and Sentinel-2 (our ground truth and expected output) products covering the same area and having their acquisition time as closest as possible to reduce feature differences. A pre-processing of the training is needed, such as co-registration, to achieve a perfect match between radar and optical data to train in the best way the Neural Network. As a preliminary case study, the area of Nile delta has been used considering the morphology of the landscape that includes deserts, urban and vegetated area, different water resources such as river and sea.

Thanks to this trained Neural Network, it is possible to achieve images in natural colours starting from almost any kind of radar images overcoming the problem of the lack cloud free optical images. Considering also to the huge amount of historical data currently preserved (e.g. ERS 1-2, ENVISAT ASAR and MERIS, Landsat just to name a few) it will be possible to apply our model to them to generate cloud free historical data sets. Future applications of the model developed, could be employed to recreate synthetically the missing information directly on optical images (due to cloud presence or others hiding phenomena) starting from radar information working on every different optical bands & wavelength.

Area C - Board 341

Spaceborne Synthetic Aperture Radar (SAR) is well-suited for broad area mapping and monitoring for a range of applications including maritime surveillance, detecting new infrastructure and changes in land cover, and gathering geospatial intelligence. To better utilize the large and ever growing archives of spaceborne SAR data, there is an increasing need for effective and automated techniques to exploit the massive datasets SAR missions provide. Deep learning (DL) has the potential to automate SAR analysis tasks, especially for mapping wide areas and for monitoring patterns of activity on a regular basis. Deep learning networks use multiple layers of nonlinear processing to learn complex patterns in the data, offering the potential for human-level performance or better on SAR exploitation tasks.

This work focuses on using MDA’s RADARSAT-2 C-band SAR sensor and state-of the-art deep learning techniques for mapping and target monitoring applications. RADARSAT-2 is a versatile sensor providing a number of different operating modes and spatial resolutions ranging from ~1-100m, 20 beam modes, 4 polarizations, and left and right look directions.

We apply SAR DL to two use cases: land cover classification and activity monitoring for geospatial intelligence (GEOINT) applications. There is increasing demand for up-to-date land cover maps, in particular identification of new infrastructure such as commercial and residential buildings and roads, to provide map updates to location-based services. SAR is also regularly used for activity monitoring over specific areas of interest such as airfields and commercial/naval ports, especially for areas often affected by cloud cover or for winter monitoring of sites located at high latitudes.

We adapt state-of-the-art DL object detection and image segmentation approaches from the computer vision domain for SAR analysis. In particular we focus on Convolutional Neural Network (CNN) DL architectures and how to modify them for SAR’s coarser spatial resolution, large dynamic range, side-looking imaging geometry and the presence of speckle noise. We evaluate the accuracy of our SAR DL architectures for different SAR resolutions, incidence angles and pre-processing such as speckle filtering. We use a cloud-based environment to train and deploy our SAR DL networks in order to process wide areas at scale and to store large volumes of data.

Area C - Board 342

Synthetic Aperture Radar (SAR) represents today one of the most powerful instruments for earth observation and environmental risk monitoring, thanks to its specific characteristics and peculiarities. The advent of new generation high resolution sensors allows imaging the earth in a deeper way, by catching much more details from the observed scenes.
This has pushed the investigation of new methodologies and new applications. However, the possibility of extracting useful information from the VHR SAR images is limited by the presence of speckle. For these reasons, in the literature of the last ten years, the filtering of VHR SAR images has become again an hot topic.
Among all the proposed techniques, the so-called Non Local (NL) approaches, have achieved the best results.
Following the recent spread of deep learning methods related to several remote sensing applications, in this work a convolutional neural networks (CNN) based solution is investigated.
The network is a fully convolutional layer network: Rectified Linear Unit (ReLU) has been used as activation function for each layer, except the last. Batch normalization has been also implemented in order to normalize the input distribution of each layer and speed up the convergence process. The weights updating rule follows the Adam optimized version of stochastic gradient descent.
Having a good despeckling algorithm is fundamental for all the post-processing applications like target and edge detection, classification and segmentation. One of most important issue of filtering is the trade-off between speckle suppression and details preservation. In order to guarantee a good balance between these two aspects, the proposed CNN tries to predict directly the speckle, working in a residual way: the ratio image, i.e. the ratio between the noisy image and the filtered one is considered. In case of a perfect filter, the ratio image should contain pure speckle. Based on this idea, a cost function for comparing the predicted speckle and the expected one is implemented.
The method is tested both on simulated and real SAR images, showing encouraging results in both cases.
In order to assess the performance, visual and numerical results are presented. As metric we use the M-index that measures the ability of suppressing speckle and preserving edge at the same time.
In order to show the capability of the method, we compared it to one of the most well-known NL solutions for despeckling, the Patch Probabilistic Based (PPB) algorithm. The results are reported in the attached Figure.
PPB has a very good ability in speckle suppression, but its result is over-smoothed. It means that in the output image some spatial details are suppressed with the speckle.
The proposed CNN is able to filter the noise showing a better details preservation than PPB: object with different dimensions are preserved as well.
In addition, in the test case, an example of a possible application (edge detection) is shown. As an input data of the detector, the noisy and the filtered (using PPB and the proposed CNN based approach) are used. The effectiveness of the proposed approach can be appreciated.
Concluding, the proposed method is very interesting and promising. The algorithm will be validated and analyzed in different test cases and for different applications.

Area C - Board 343

The typical speckle pattern in detected Synthetic Aperture Radar (SAR) images is generated by the interaction between the transmitted coherent e.m wave and the backscattering surface.
Speckle carries useful information about the properties of the illuminated area. As an example, the speckle phase variation connected to a change in the radar looking angle can be exploited to measure the relative elevation of the illuminated area (SAR Interferometry).
However, in many applications where only the amplitude of a SAR image is of interest, speckle acts as noise. Objects detection, flood mapping and land use classification using SAR images are just some of the applications that benefit from speckle noise reduction.
In principle, the best speckle filter should reduce the amount of speckle by estimating the mean backscattered value from all pixels belonging to homogeneous regions while preserving edges and individual features in heterogeneous areas. Typically, in presence of man-made structures or any other targets characterized by strong pointwise scatterers (e.g. Permanent Scatterers [1]), we have almost no speckle and the backscattered value should be left as it is.
The main difficulty in reaching this goal is the object segmentation of a single SAR image. The best object segmentation results can be achieved by using a stack of multi-temporal or multi-baseline images as done for example with the MuLoG [2] and the SqueeSAR [3] approaches, provided that the reflectivity of the area of interest is not affected by major changes. The speckle reduction results achievable by such multi-image techniques should be considered as a reference for any speckle filtering approach working on single SAR images.
Many local (e.g. Lee, Frost) and non-local (Wavelet, PPB) algorithms have been proposed to improve speckled image quality since the 80’s. Very recently with the rising scientific and commercial interest to the field of machine learning, the problem of a single SAR image de-speckling has been approached by means of artificial intelligence techniques [4][5][6]. In general, multi resolution convolutional neural networks (CNN) can successfully handle problems that can be easily solved by human and are very hard to be solved with classical signal processing approaches. In the present work we adapted to SAR images de-speckling a convolutional network architecture (U-net [7]) that has been designed for biomedical image segmentation. The results achieved on hundreds of Sentinel-1 images over different kind of terrains clearly show the great potential of CNN when applied to speckle reduction, generating images quite similar to “multi-image reflectivity maps” typical of multi-image approaches.


[1] Ferretti, A. Prati, C., Rocca, F. (2001), “Permanent Scatterers in SAR Interferometry”, IEEE Transactions on Geoscience and Remote Sensing, 39 (1), 8- 20.
[2] Charles-Alban Deledalle, Loïc Denis, Sonia Tabti, Florence Tupin, (2017), “MuLoG, or How to apply Gaussian denoisers to multi-channel SAR speckle reduction?”, IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4389-4403, 2017
[3] Ferretti, A., Fumagalli, A., Novali, F., Prati, C., Rocca, F. and Rucci, A. (2011), “A New Algorithm for Processing Interferometric DataStacks: SqueeSAR”, IEEE Transactions on Geoscience and Remote Sensing, 49 (9), 3460- 3470.
[4] Puyang Wang , He Zhang ,Vishal M. Patel , S., (2017) “SAR image despeckling using a convolutional neural network”, IEEE SIGNAL PROCESSING LETTERS, VOL. 24, NO. 12.
[5] Chierchia, G., Cozzolino, D., Poggi, G., & Verdoliva, L. (2017, July). SAR image despeckling through convolutional neural networks. In Geoscience and Remote Sensing Symposium (IGARSS), 2017 IEEE International (pp. 5438-5441). IEEE.
[6] Zhang, Q., Yuan, Q., Li, J., Yang, Z., & Ma, X. (2018). Learning a dilated residual network for SAR image despeckling. Remote Sensing, 10(2), 196.
[7] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.

Area C - Board 344

Landslide processes are a major hazard concerning mountainous regions. Mapping unstable slopes is crucial for an effective hazard assessment, hence updated and reliable landslide catalogs are essential. The procedures for generating these catalogs are, however, mainly dependent on human-based interpretations. Depending on the extension of area of interest and available data, this procedure is very slow and usually biased. Fortunately, the constantly growing fleet of Earth Observation (EO) systems provides high-resolution information and repeat coverage required for mapping of unstable slopes. Automatic processing of this EO Big Data can reduce the dependency from human involvement, thereby increasing the frequency of updates to landslide catalogs.
In this work, we study machine learning strategies for the analysis of slope instabilities at a regional scale. The idea is to reduce dependency from slow and subjective human-interpretation, which in turn will facilitate with the generation of quick, consistent and reliable landslide inventories. The available archive of EO datasets and landslide inventories will be used to train machine algorithms for automatic classification of stable and unstable slopes. We compare the effectiveness of individual and ensemble learners in object-based image analysis (OBIA) system. Deviating from the conventional practice of image segmentation from optical imagery, we introduce a new approach to defining objects by grouping the aspect values in digital elevation models (DEM). This pushes the definition of objects to resemble more like a hill-slope face and is a better representation for slope stability classification problem. We further identify key features which are critical in the determination of the object’s stability. A case study is presented for a region in the Himalayan range of North-Eastern Bhutan. A complete inventory of landslide was manually mapped by integrating the information from a 5-meters DEM, high-resolution optical images, and radar interferometry. This mapped inventory had a hold-out set for validation, while the rest was used for training the learning algorithms. Here we present the initial results and observation from this work.
This work is done in the framework of European Commission's Horizon 2020 project "BETTER", with the main objective to facilitate the usage of large volume and heterogeneous datasets by downstream users, so that they can focus on the analysis of the extraction of the potential knowledge within the data and not on the processing of the data itself. More information is available on the website https://www.ec-better.eu/.

Area C - Board 345

With the wealth of EO data available today, focus has shifted to automated image processing and information extraction to maximize the use of the available big data. The high temporal and spatial resolution of these data provides an opportunity for continuous natural hazard recognition and analysis, in particular for landslide and rockfall hazards. Detection and classification of objects like rockfalls in available EO images can be performed by Deep Neural Networks (DNNs). These networks can be trained and subsequently deployed to automatically detect and map landslides and rockfalls in satellite imagery on regional or even global scales. Due to the EO image acquisition frequency, the resulting landslide distribution maps would be available not only on large spatial scales, but with high temporal regularity. Time-series analysis of such feature distribution maps would reveal the spatial distribution of landslide hazards, point to seasonal or atmospheric effects on landslide occurrence, or provide a risk estimate for regions to be affected by this type of natural hazard. However, difficulties have been encountered for Deep Learning applications for landslide detection on Earth due to vegetation, atmospheric effects, seasonal differences, or lack of spatial resolution & coverage, currently impeding a successful implementation of DNNs for automated landslide detection on Earth.
On the Moon, fewer environmental influences are present and the performance of DNNs for automated slope instability detection on large spatial scales can be properly assessed. We implemented and validated a single-stage Convolutional Neural Network to automatically detect rockfalls on the Moon in the available high-resolution satellite imagery. The network reaches recall values between 44 and 98 %, as well as precision values between 45 to 100 %, depending on the applied network confidence threshold, and is capable of exploiting the currently available data archive with more than 1 million images to produce large-scale rockfall distribution maps. However, even with fewer environmental influences as encountered on Earth, we find that automatization of lunar slope instability mapping is challenging and complex. Potential sources of error are inconsistent data quality, variations in illumination conditions, insufficient amount of training data, technical limitations such as spatial resolution in relation to the size of the object of interest, and the natural environment itself. The lessons learned “in the sandbox” on the Moon could be beneficial for the endeavor of the community to develop new data processing methods for EO with DNNs and machine learning in general.

Area C - Board 346

Drifting icebergs can threaten navigation and marine operations and are prevalent in a number of regions that have active offshore operations. Satellite synthetic aperture radar (SAR) is naturally applicable to map and monitor icebergs and sea ice due its ability to provide images day or night, through cloud or fog, and various wind conditions. There are several notable examples of its use to support operations and shipping, including for example the Canadian Arctic, the Grand Banks, the Southern ocean, the Barents Sea, offshore Greenland and the Kara Sea. Increasing prevalence of SAR constellations are making data ubiquitous and easier to access for iceberg surveillance.

New constellations of satellites present a new paradigm for ice surveillance in persistence, reliability and cost. To fully extract the value of the data from these constellations, automation and cloud-based processing must be implemented. This will allow more timely and efficient processing, lowering monitoring costs by at least an order of magnitude. The increase in data persistence allows large regions to be monitored daily for ice incursions, thus lowering operational risk and costly diversions. The increased prevalence of these data present a requirement for automation to facilitate the efficient and timely processing of these data for iceberg detections.

The process of automating SAR-based iceberg surveillance involves the creation of a process flow that is robust and requires limited human intervention. The process flow involves land-masking, target detection, target discrimination and product dissemination. Land masking involved the removal of high clutter land from the imagery to eliminate false detection from these locations. Target detection usually involves an adaptive threshold to separate true targets from the background ocean clutter. A constant false alarm rate (CFAR) is a standard technique used in radar for this purpose. Target discrimination involves an examination of the distinct features of a target to determine if they match the features of icebergs, vessels or other ‘false alarms’ (e.g., marine wildlife, clutter). The final stage is the production of an output surveillance product, which can be a standard iceberg chart (e.g., MANICE) or something that can be ingested into a GIS system (e.g., ESRI shapefile, Google KML).

The target discrimination phase is one of the most important stages because it provides feedback to operations as to the present of targets of interest (icebergs and vessels). The authors have used computer vision techniques successfully to train target classifiers. Standard techniques usually result in classifier accuracies of between 85%-95%, depending on the resolution of the SAR (higher resolutions produce more accurate results) and the availability of multiple polarizations. To see if new machine learning techniques could be applied to increase classifier accuracy, a dataset of 5000 ship and iceberg targets were extracted from Sentinel-1 multi-channel data (HH,HV). The imagery were collected in several regions (Greenland, Grand Banks, and Strait of Gibraltar). Validation either came by way of supporting information from the offshore operations, or inferred by location. An online machine learning competition was hosted by Kaggle, a company that conducts online competitions on behalf of their clients. The detection data were made available by Kaggle to the broad internet community. Kaggle has a loyal following of data scientists who regularly participate in Kaggle competitions. The competition was hosted over a three month period; over 3300 teams participated in the competition. The competition produced an improved classifier over standard computer vision techniques; the top three competitors had 4-5 stageclassifiers that increased classification accuracy by approximately 5%.

Area C - Board 347

Automatically generated high resolution sea ice maps have the potential to increase the use of satellite imagery in arctic applications. Applications include marine navigation, offshore operations, validation of ice models, and climate research. Especially for arctic marine navigation, frequent ice maps in high resolution are requested by most users, as documented by an internal project
stakeholder survey.
We present current results from our large-scale study of high resolution ice maps generation with Convolutional Neural Networks (CNNs). Our study is based on dual polarized (HH+HV) Extra Wide swath (EW) SAR data from the Copernicus Sentinel 1 satellite mission and we generate pixel-wise sea ice estimates in 40m x 40m resolution. The presentation will include a model validation against expert annotations of SAR images. In the near future we will expand our study to include AMSR2 Microwave Radiometer (MWR) data as input. The addition of MWR data can potentially solve the ambiguities in SAR data over open water, due to SAR backscatter variation at different wind conditions. Some CNN estimates are observed to confuse very homogeneous ice surfaces with similar backscatter open water scenarios, but results show a clear potential for this methodology.

Our work is carried out under a Danish research project named Automated downstream Sea Ice Products (ASIP). The project goal is to automate generation of sea ice information from satellite images. ASIP is a collaboration between the Danish Meteorological Institute (DMI), the Technical University of Denmark and Harnvig Artic and Maritime. It sets out to automate, partially or fully, the extraction of arctic sea ice information from satellite imagery. Today, ice mapping is mainly done manually by ice-experts at national Ice Centers around the world. The project goal will enable analyzing larger quantities of satellite data, for better utilization of the available Sentinel-1 images and for providing ice maps to users more frequently. Recent literature shows an increased interest in algorithms for extraction of sea ice information, [2, 4]. As a part of the ASIP project a thorough analysis of the need for ice information was carried out among users by Harnvig Arctic and Maritime. This resulted in "ASIP Internal Stakeholder Survey Report", which substantiates the specic needs. One of the conclusions from this report is that 90% of use cases need simple ice/no-ice information for marine route planning purposes in high resolution (< 250m pr. pixel). Meeting this resolution requirement is unfortunately not possible with current MWR data alone, though its properties are otherwise good for ice concentration estimations. Hence, SAR data is the
only source with regularly coverage as input data.

A training data set was prepared by selecting all regional Ice Charts that were manually produced by the ice analysts in the DMI Ice Service based primarily on Sentinel-1 SAR imagery. SAR imagery has proven very suitable for Arctic sea ice monitoring, due to the radar sensors capability to see through clouds and in polar darkness. The training data set consists of a match-up of more than 900 ice charts and corresponding SAR imagery covering the period from the availability of the Sentinel-1A sensor data in November 2014 and Sentinel-1B data from September 2016, until December 2017. The individual ice charts cover dierent areas (depending on users and season) but the data set spans all of Greenland coast, apart from the northernmost regions, and with denser
coverage in Southern Greenland.

The CNN architecture we use in this work is inspired by models from image segmentation [1]. We train it on 400 000 sub-images of 250x250 pixels cut out from Sentinel-1 scenes corresponding to more than 25 billion pixel samples. Labels for each pixel are taken from ice concentration parameter in the ice-charts. Our initial approach has simply been to threshold ice concentrations,
and consider <10% as open water and otherwise ice. This approach turns the problem into a pixelwise classication task, i.e. segmentation, and our CNN model produces a 250x250 segmentation mask for each input image. The way ice charts are produced combined with our threshold approach introduces an amount of mis-labelled pixels, but Deep Neural Networks (DNNs) are known to cope with "noisy" labels [3].

Current results can be seen in Figure 1, which shows the sub-images in a Sentinel-1B scene from June 2017 that matches with an DMI produced Ice Chart. It should be noted that labels are very coarse compared to the level of detail in a Sentinel-1 image. Despite this, the CNN still correctly classies the lead of water running into the ice in Figure 1 (marked with a redcircle).

Acknowledgements:
We would like to thank our funding partner Innovation Fund Denmark and the NVIDIA Corporation for supplying the GPU Cards used for training our model.

References
[1] Liang-Chieh Chen, George Papandreou, Florian Schro, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
[2] Juha Karvonen. Baltic sea ice concentration estimation using sentinel-1 SAR and AMSR2 microwave radiometer data. IEEE Transactions on Geoscience and Remote Sensing, 55(5):2871{2883, 2017.
[3] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 843{852. IEEE, 2017.
[4] Lei Wang, K Andrea Scott, and David A Clausi. Sea ice concentration estimation during freeze-up from SAR imagery using a convolutional neural network. Remote Sensing, 9(5):408, 2017.

Area C - Board 348

There is an overwhelmed amount of data from Copernicus satellites. For instance, thousand images are produced every day representing a daily average of 3,45 TB of S1a/S1b published (ESA Sentinel Data Access Annual report 2016). A significant amount covers ocean surface, used for a wide range of applications involving public and private stakeholders. Notwithstanding the current great impact of Sentinel-1 images for the oceanographic community, do we really exploit the full imaging capabilities of these C-band SAR data acquired over ocean surface?

Beyond wind field measurements, sea ice monitoring or oil spill detection, a set of metocean features are well observed by S-1 sensors. To name a few, atmospheric fronts, oceanic fronts, rain cells, micro convective cells, internal waves, gravity waves, biologic slicks, upwelling or wind streaks are phenomena of potential interest for many end-users while being generally discarded in the SAR images. For sciences academic and industries, new research perspectives and new potential applications/services could be opened if those mechanisms are automatically and systematically tagged. In addition, flagging of these features could also benefit to current products/services providing better data quality. With recent progress in computer vision thanks to deep learning approaches in conjunction with the rise of large database and higher computing power, these flagging activities are now possible.

The image processing community started a decade ago with the ImageNet database which is a large visual database (14 million images) designed for use in visual object recognition software research. 20000 categories have been hand-annotated indicating the presence of objects within image, providing also bounding boxes for some them. Launched in 2010, ImageNet Large Scale Visual Recognition Competition (ILSRVC) is an annual competition where research teams evaluate their image processing algorithms on the ImageNet data set. Convolutional Neural Network (CNN) deep learning (DL) framework soon appeared to be the most effective technique. Alex-Net, U-Net, Fully Convolutional Network (FCN) and VGG were some of the proposed CNN framework in the past years. In this fast-evolving domain, Deeplabv3+ is currently one of the most powerful architecture enabling semantic image segmentation.

Through the inspiring advances of CNN/DL applied to object recognition in image, we have implemented CNN approaches on Copernicus S-1 SAR images acquired over the open ocean. For the very first time (as far our knowledge goes – sea ice classification and iceberg vs vessel have been already investigated), DL frameworks are tested in the aim to provide a semantic segmentation of Sentinel-1 scenes acquired over open ocean surface, with the final objective to tag all these metocean features which have been neglected for years.

More specifically, a database of 37k Wave Mode (WM) imagettes has been annotated with 10 different classes (Pure Ocean Swell, wind streaks, micro convective cells, rain cells, biological slicks, sea ice, icebergs/bright targets, low wind area, atmospheric fronts, oceanic fronts). It is then used as training database in Inception V3 framework. Fine-tuned model provides cross-validation and test accuracy of more than 95%. To provide some insights on the classification results, T-distributed stochastic neighbor embedding (T-SNE) algorithm is used to project 10-dimensional predicted vector to a 2D map. This enables to highlight co-occurrence of phenomena, errors in the labelled database, and mixture/continuum of geophysical phenomena (for instance micro-convective cells / wind streaks).

As our initial training database provide only one label per imagette, we adopted a sliding window approach to attempt the semantic segmentation of oceanic SAR images. Sub-imagettes are extracted using either a random approach or a method based on the peak location of the Class Activation Map (Selvaraju et al. 2016) depending on whether the geophysical phenomena is localized or not on the original WM imagette. In this presentation, we discuss the implementation of the methodology for large swath S-1 images (IW and EW modes). The potentials and challenges of using DL with SAR images for oceanography will be outlined.

Area C - Board 349

Radar altimetry is commonly used for monitoring changes within the cryosphere and its contribution to sea-level rise. The revolutionary design of CryoSat-2 (CS2) features SAR and interferometric capacities that allow us to increase spatial resolution while resolving the angular origin of off-nadir echoes occurring over sloping terrain, making it particularly adapted to monitor ice-sheet margins. On top of measuring more precisely the elevation of Point-Of-Closest-Approach (POCA), the standard level-2 product, CS2 SARIn mode allows the CS2 Swath SARIn approach. This produces elevations beyond the POCA, leading to between 1 and 2 orders of magnitude more measurements than level-2 products and has enabled fine 500m spatial resolution surface elevation models from each satellite pass.

However, there is variability in radar elevation estimates, often due to the penetration of radar waves into snow and firn, yielding differences compared to local airborne LiDAR altimeters which are less impacted by penetration. Local Airborne LiDAR campaigns however have limited coverage and have temporal restrictions due to flights only operating in favourable weather conditions. Therefore, a way to utilise the accuracy of local airborne LiDAR together with the spatial and temporal advantages of using CS2 Swath data is desirable. While neural networks are increasingly being used in a wide variety of domains to enhance predictions and detect changes, they have not previously been applied to radar altimetry to correct for elevation biases within the cryosphere.

In this study, we present a novel approach to adjusting for elevation bias by creating a two-layer neural network that is trained to predict the elevation provided by local airborne LiDAR from CS2 Swath data where both data are available and then apply that model where only CS2 Swath data are present. By using a neural network we remove the need for an individual to build a model that defines the relationships between input variables; of which many relationships are largely unknown. A neural network overcomes this by automatically determining the relationships between variables in its hidden layers during the training cycle, capturing behavioural patterns that are not currently known.

We investigate the challenges with building such models and review a variety of configurations and considerations such as overfitting and overgeneralising. Finally, we present two proof of concepts that show good spatial and temporal transfer ability across three topographically different study areas in Greenland (Jakobshavn, Storstrommen and the South East). These models compensate for 70-90% of the mean penetration by radar in the CS2 Swath data, while reducing the root mean squared error by 10-17%.

Area C - Board 350

Recent launches of high-resolution satellite sensors mean Earth Observation data are available at an unprecedented spatial and temporal scale. As data collection intensifies, our ability to inspect and investigate individual scenes for harmful algal or cyanobacterial blooms becomes limited, particularly for global monitoring. Algal Blooms and River Plumes are visible to trained experts in high resolution satellite imagery from Red-Green-Blue composites. Therefore, computer-assisted detection and classification of these events would provide invaluable information to experts and the general public on everyday water use.

Advances in image recognition through Deep Learning techniques offer solutions that can accurately detect, classify and segment objects across thousands of images in a fraction of the time a human operator would require, while inspecting these images with much greater detail. Deep Learning techniques that jointly leverage spectral-spatial data are well characterised as a solution to land classification problems and have been shown to be accurate when using multi- or hyper-spectral data such as collected by the Sentinel-2 MultiSpectral Instrument.

This work develops on state-of-the-art natural image segmentation algorithms to evaluate the capability of Deep Learning to detect and outline the presence of Algal Blooms or River Plumes in Sentinel 2 MSI data. The challenges in the application of these approaches are highlighted in the availability of suitable training and benchmark data, the use of atmospheric correction and neural network architecture design for utilisation of spectral data.

Current Deep Learning network architectures are evaluated to establish best approaches to leverage spectral and spatial data in the context of water classification. Several spectral data configurations are used to evaluate competency and suitability for generalisation to other Optical Satellite Sensor configurations. The impact of the atmospheric correction technique applied to data is explored to establish the most reliable data for use during training and requirements for pre-processing data pipelines. Finally a training dataset and associated Deep Learning method is presented for use in future work relating to water contents classification.

Area C - Board 351