Timely and accurate information on forest above-ground biomass (AGB) is required for understanding carbon balance, future climate, the sustainability of current politics and the emerging bioeconomy. Optical Earth Observation (EO) in the visible to short infrared spectral region can contribute to this challenge. This is supported by the more frequent and higher quality EO data, mainly contributed by the new Sentinel-series satellites developed and launched by the European Space Agency (ESA). The new satellites deliver data with better spatial and spectral resolutions in a rapidly increasing volume, calling for new processing algorithms. Current algorithms fall short in making a full use of the higher information content of the new data, and have trouble handling the data volume produced by current and forthcoming satellite instruments.

Artificial intelligence (AI) has shown promise to overcome many of the shortcomings in existing empirical methods for retrieving forest characteristics, such as its structure and biomass, from EO data. It can also handle efficiently and rapidly large data volumes. However, despite optimistic case studies, no large-scale applications using optical EO for biomass retrieval have emerged. Up to now, AI has been applied to simple canopies with little structure and no woody biomass.

Recent developments in deep learning have made it feasible to utilize it for analyzing complex structured vegetation with significant AGB storage such as the boreal forest. We hypothesize that with a carefully selected approach, deep learning approaches can retrieve the forest AGB which is indirectly, but physically, related to the structural parameters affecting forest reflectance, e.g., crown volume and shape, or tree density.

We propose to use a well-validated physically-based forest reflectance model to simulate the spectral reflectance factors of all possible European boreal forest canopies under different illumination conditions. The model will be used to simulate the reflectance signatures of a wide range of forests with structural parameters taken from forestry databases in Sweden, Finland, Estonia and Russia. We will use the simulated reflectance signatures, corresponding to satellite-measured signals, to train a combination of deep learning algorithms with several desired properties such as :
- Recurrent Neural Networks or Long Short-Term Memory (LSTM) networks for regression
- Fully Convolutional Networks (FCN) or Convolutional Neural Networks (CNN) for their ability to extract hierarchical visual features from multispectral images
- semi-supervised approaches to deal with the limited availability of labelled training data
- deep probabilistic approaches to provide confidence intervals on the estimated outputs

Once trained, we will apply these algorithms to optical EO data from the European boreal zone, and hyperspectral data from Finland. The AI retrieval results will be compared against forestry data from test sites in each of these regions. We will also investigate the potential of Generative Adversarial Networks (GANs) to augment the available reference forest data and generate more diverse and realistic images than from simulations based solely on physical models.

AIROBEST will provide a new look on our forests on a continental scale. It will deepen the apprehension of AI methods in Earth Observation and our understanding of the changes in environment. The algorithms produced in the project will be integrated with other EO-based forest monitoring tools developed at VTT, creating a capacity for comprehensive forest information retrieval for research, governmental bodies, and industry.

This study is part of the AIROBEST project on Artificial Intelligence for Retrieval of Forest Biomass & Structure, funded by the Finnish Academy (grant 317387) as part of the programme on Novel Applications of Artificial Intelligence in Physical Sciences and Engineering Research (AIPSE).

Forests play a crucial role in the global carbon (C) cycle by storing and sequestrating a substantial amount of C in the terrestrial biosphere. Due to temporal dynamics in climate and vegetation activity, there are significant regional variations in carbon dioxide (CO2) fluxes between the biosphere and atmosphere in forests that are affecting the global C cycle. Current forest CO2 flux dynamics are controlled by instantaneous climate and vegetation conditions as well as memory effects related to seasonal dynamics, disturbances and extreme climate events. Our level of understanding of the legacies of these processes on net CO2 fluxes is still limited due to their complexities and their long-term effects, and therefore may be improved through better modeling that include memory effects. Here, we combined remote sensing, climate, and eddy-covariance flux data to study net ecosystem CO2 exchange (NEE) at 185 forest sites globally. Instead of commonly used non-dynamic statistical methods (e.g. decision trees), we employed a type of recurrent neural network (RNN), so-called Long Short-Term Memory networks (LSTMs), that characterize and extract information from the temporal dynamics of vegetation and climate time series at a particular site. The resulting data-driven model integrates interannual and seasonal variations of climate and vegetation dynamics by using Landsat and climate data at each site. We showed that an LSTM algorithm was able to effectively describe the overall seasonal variability (Nash-Sutcliffe efficiency, NSE = 0.66) and across-site (NSE = 0.42) variations in NEE, while it had less success in predicting specific seasonal and interannual anomalies (NSE = 0.07). Our results also demonstrated that a LSTM approach with embedded climate and vegetation memory effects outperformed a non-dynamic model (i.e. Random Forest) for estimating NEE, particularly at the seasonal scale. Additionally, we investigated the temporal features of vegetation that are relevant for estimating NEE via a LSTM-based factorial experimental design. We showed that the vegetation mean seasonal cycle embeds most of the information content to realistically explain the spatial and temporal variations in NEE. These findings show the relevance of capturing memory effects from both climate and vegetation in quantifying spatio-temporal variations in forest NEE.

Deep learning has been used successfully for computer vision problems, e.g. image classification, target detection and many more. We use deep learning in conjunction with ArcGIS to implement a model with advanced Convolutional Neural Networks (CNNs) for lithological mapping in the Mount Isa region (Australia). The area is ideal for geological remote sensing as there is only sparse vegetation. Besides freely available Sentinel-2 and ASTER data, several geophysical datasets are available from exploration campaigns. There are first studies based on Sentinel-2 data for geological applications and ASTER has a long history in lithological mapping, however not in the field of deep learning. By fusing the datasets and thus covering a wide spectral range as well as capturing geophysical properties of rocks, we aim at improving classification accuracies. This approach can also be used for other applications that aim at classifying landcover.
We developed an end-to-end deep learning model with 2-D convolutional and deconvolutional layers using Tensorflow. Our model was inspired by the family of U-Net architectures, where low-level feature maps (encoders) are concatenated with high-level ones (decoders), which enables precise localization. This type of network architecture was especially designed to effectively solve pixel-wise classification problems as is the case for lithological classification. We spatially resampled, reprojected and fused the multi-sensor remote sensing data into an image cube as input for our model. The connection between ArcGIS and the deep learning libraries was achieved by using the Python API for ArcGIS and by implementing the training and testing workflow into Jupyter Notebooks. Our model classifies each pixel of the multiband imagery into different types of rocks according to a defined probability threshold.
After fitting our model on 50 million pixels, results show that the overall accuracy based on Sentinel-2 bands alone is around 73% and can be improved by adding ASTER bands to almost 75%. By including both, ASTER and geophysical data, results could not be improved.
In conclusion, by fusing Sentinel-2 and ASTER spectral bands that perfectly capture major absorption features of clay minerals and mafic minerals such as pyroxenes and carbonates, we achieve rather good classification results for the different lithologies, even though there are many labelling errors in the training data.

The first few hours after a natural disaster strikes are critical. Identifying the most vulnerable and severely-affected areas is crucial to design an effective response plan. In the past few years imagery captured by Unmanned Aerial Vehicles (UAVs) has played a key role in assessing areas that are difficult to access or performing the initial search in a rescue phase as the images collected by UAVs can be used to assess the overall level of damage inflicted on the distressed areas as well as the most effective allocation of the limited relief resources by humanitarian organizations.

Analyzing large volumes of high-resolution aerial images generated after a major disaster remains a challenging task in contrast to the ease of acquiring them due to quick deployment and low operational costs of UAVs. A popular approach is to use a hybrid strategy where initially a crowdsourcing effort is carried out to create a labeled training set which is used to build a machine learning model [1]. The trained model then automatically analyzes the incoming images. However, until now, the object recognition and localization tasks suffered from low accuracy and lack of robustness in an uncontrolled environment due to the challenges associated with the images acquired by UAVs in disaster settings.

In this work, we explore how most recent deep learning techniques in general object recognition and localization can be helpful in localizing and identifying damaged structures during disasters. Specifically, we evaluate two of the state-of-the-art deep learning models, i.e., Faster R-CNN [2] and YOLOv2 [3], to locate the built structures and assess their damage severities in UAV images.

The imagery used in this work was captured by UAVs as part of the relief efforts in the aftermath of the devastating Cyclone Pam in the Pacific Ocean during 2015. To annotate the collected images, we supervised a group of volunteers to draw polygons around areas of built structures and to assess the severity of damage into three categories: mild, medium and severe. The dataset consists of 1,085 images and every image contains one or more levels of damage. Each image was labeled by two or more annotators and finally a global annotation was produced using a union with majority voting. The resulting label distribution is mild (54%), medium (22%) and severe (24%).

The average bounding boxe size is approximately 430x275 pixels. Interestingly, this average bounding box size is almost equivalent to the average image size in the PASCAL-VOC dataset [4]. There are some images in the dataset where built structure can cover the entire image, or the scale ratio of the largest to the smallest bounding box can be more than 100 folds. On the other hand, most of the images contain on average 5 damage structures while few images may also have as high as 20 or more. The change in the number of objects observed in an image happens when the UAV reaches higher altitude; and with change in viewpoint and altitude, it becomes more difficult to assess the damage severity of structures. This is unlike the images available in the popular computer vision benchmark datasets such as ImageNet [5] and Places2 [6].

Although there are some small scale aerial imagery datasets widely used in the remote sensing domain, mostly for the image classification task, such as UCMerced [7] and RS19 [8], the state-of-the-art results on these small datasets have already reached almost perfect performance [9-10]. Therefore, more challenging aerial imagery datasets such as the one presented in this paper are needed for the scientific advancement in computer vision and remote sensing domains.

We performed extensive experiments to assess the accuracy of the aforementioned object detectors, i.e., Faster R-CNN and YOLOv2, on damage assessment task. We used pre-trained network weights and did a full sweep to find the best parameter set for our UAV dataset via a four-fold cross-validation. Finally, the results are presented on a separate test data.

For Faster R-CNN, the two important parameters are scale and aspect ratio. These parameters highly depend on the range of bounding box sizes. We computed average bounding box size for our data and ran through a range of scale and aspect ratio combinations. Since we have bounding boxes with comparatively more width than height and high scale variance, we get best results with values of aspect ratio and scale as {1, 1.5, 2, 2.5} and {64, 128, 256, 512}, respectively. We also tried with different RPN overlap ratio but default setting of 0.7 did not lose any further performance.

Unlike Faster R-CNN, YOLOv2 learns from a range of input sizes while training in an end-to-end manner. Picking up better priors for bounding boxes to start with makes it easier for the network to learn to predict better detections. To do this, we run K-means clustering on the training set bounding boxes to automatically find better priors. This is an important parameter as it would vary with the type of data set. We also found K = 5 as used in the original paper to be a good tradeoff between model complexity and performance.

We compare the performance of both object detectors in terms of their mean average precision (mAP). Faster R-CNN outperforms YOLOv2 as we obtain the best mAP of 28 for Faster R-CNN and 15 for YOLOv2 detector models in comparison to mAP of 70.4 (Faster R-CNN) and 73.4 (YOLOv2) on PASCAL VOC dataset. The lower performance is due to the fact that our task is more challenging than recognizing object categories in PASCAL VOC. Further analysis showed that on average 60 additional objects (buildings) were detected by the algorithm but due to missing ground truth annotations they cannot be evaluated. Also, a high percentage of medium (50%) and severe (60%) structures remain undiscovered.

Attached figure shows few examples to compare Faster R-CNN and YOLOv2 in more detail. (a) exemplifies a case where there is confusion between construction and actual damage. The building closer to the camera covers better part of the image and features are more descriptive than one small building of the same type. Difference in viewpoints play a vital role in these situations. Also, Faster R-CNN detects an additional object not being labeled. (b) illustrates simple case where both detectors perform equally well. (c) is again a good case where detections are correct and also both detect unannotated objects. (d) and (e) show difficult cases where images are taken at higher altitudes and understanding their damage level is even challenging for humans. Also, these examples show that Faster R-CNN is better than YOLOv2 in detecting smaller objects. (f) shows an interpretation error, it could be rubbles but it is equally true that it can be a destroyed building. (g) highlights the scenario where there is agreement on some damage but detector gets the type wrong. There are cases where medium and severe damage gets confusing because it is quite subjective. (h) exemplifies the inherent labeling task noise because instead of annotating just the damaged portion entire roof is annotated. This reduces the impact of damage as large portion of the bounding box contains no damage and the average feature learned is more "mild" than it actually is. All these cases show that both detectors are quite similar but Faster R-CNN is better at detecting small objects than YOLOv2.

The results presented in this work indicate that state-of-the-art vision-based algorithms may assist human evaluators in detecting and determining damaged structures. However, more powerful and task-specific models need to be tailored for automated damage assessment from UAV images during disasters.

[1] F. Ofli, P. Meier, M. Imran, C. Castillo, D. Tuia, N. Rey, J. Briant, P. Millet, F. Reinhard, M. Parkan, and S. Joost. Combining human computing and machine learning to make sense of big (aerial) data for disaster response. Big Data 2016.

[2] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

[3] J. Redmon, and A. Farhadi. YOLO9000: Better, Faster, Stronger. CVPR 2017.

[4] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV 2010.

[5] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV 2015.

[6] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million Image Database for Scene Recognition. PAMI 2017.

[7] Y. Yang and S. Newsam. Bag-of-visual-words and spatial extensions for land-use classification. SIGSPATIAL 2010.

[8] G.-S. Xia, W. Yang, J. Delon, Y. Gousseau, H. Sun, and H. Maitre. Structural high-resolution satellite image indexing. ISPRS 2010.

[9] M. Castelluccio, G. Poggi, C. Sansone, and L. Verdoliva. Land use classification in remote sensing images by convolutional neural networks. arXiv 2015.

[10] K. Nogueira, O. A. Penatti, and J. A. dos Santos. Towards better exploiting convolutional neural networks for remote sensing scene classification. Pattern Recognition 2017.

With the ease of availability of Earth Observation (EO) data, both temporal and spatial, the use of data-driven machine learning approaches to extract information from satellite images is growing continuously. This has led to an increasing interest in introducing EO-based services into the line of business of various sectors that could benefit from the integration of EO data with current technological improvements in terms of big data handling, available computing power and, of course, artificial intelligence. More specifically, the ongoing advancements in deep learning and exemplary results obtained in a variety of fields has increased their popularity within both the research and business communities. The RNN architecture, commonly used in Natural Language Processing, has become a common choice to handle temporal sequences. The deep learning models have the capability to learn complex features from the dataset available. The mandatory requirement is clean and reliably labeled data, which can be a challenging task in the big data problems. What if the labels are not reliable due to manual errors or due to the complexity of annotating data?

In this work, the main ambition is to deal effectively with the problem of crop type identification and classification for coping with specific business needs of the agricultural sector. In a nutshell, the aim is to classify agricultural parcels, against the crop type that is being cultivated in them, based on images that derive from Sentinel-2A satellites. The satellite images from different dates in the year 2016 together with parcel location and crop variety form temporal sequences that are used as data for training the models. What is of utmost importance, is to design a methodological pipeline (in the figure) to verify the reliability of the data with minimum supervision and then, use this clean data to train the model for the classification problem at hand. The proposed method is a complete pipeline for cleaning data using minimal supervision and subsequently, creating a crop specific pixel-level classification model. Instead of using traditional methods based on data specific heuristics and hand-crafted filters to identify outliers, a neural network was trained to remove the unreliable data. An RNN-based autoencoder model, motivated from the research by Mou et al. [1] and Russwurm et al. [2], was trained for each crop type. The spectro-temporal features are obtained for the input data and analyzed by means of k-means clustering. The clusters in the feature space separate reliable data from the potential outliers. An expert can further refine these data groups by annotating boundary cases. In this manner, the effort of the expert is optimized by focusing in targeted areas. Having selected the appropriate data, a single variety pixel-level classifier model (similar to Mou et al. [3]), was trained and used for detecting whether an area contains a specific crop type or not. Experiments on different crop types have shown excellent results in identifying the crop variety, while reducing the time and effort required for model creation. Given sufficient data from a multitude of crop varieties, a multi-class model can also be trained with more complex sequential encoders [4]. This approach can be exptended further to integrate ground sensors, aerial and ground imagery in order to achieve a finer spatio-temporal resolution.
The described pipeline provides support for two agricultural pilot activities in DataBio [5], namely the Insurance and CAP Support pilots in Greece. Both of these pilots require EO-based tools for continuous monitoring with the ability to support crop status and type assessment checks respectively. NEUROPUBLIC is leading these pilot activities, serving also as the main component and data provider. CSEM will offer land usage classification/identification as a service through the DataBio platform, thus, allowing its evaluation in already operational business case scenarios (supporting DataBio’s vision for interchangeable and interoperable components).
Using the proposed RNN based approach for detection and classification of crop types creates a new possibility in handling Big Data in the domain of Earth Observation with a multitude of potential applications in the areas of precision and smart agriculture, pre-emptive forest management, health monitoring, and damage assessment. The previously described process of data selection and model creation offers a structured method of digitizing expert knowledge in a data-driven architecture offering a viable operational solution for machine learning integration in EO based services. The presented pipeline significantly reduces the complexity of creating models by removing the need for handcrafted filtering, and making it a cost-effective option for bringing neural network models to the market. This work has received funding from the European Union’s Horizon 2020 research and innovation programme within the project DataBio focusing on using Big Data for the bioeconomy under grant agreement No 732064.


[1] L. Mou, P. Ghamisi and X. X. Zhu, "Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network for Hyperspectral Image Classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 56, pp. 391-406, 2018.
[2] M. Russwurm and M. Koerner, "Multi-Temporal Land Cover Classification with Long Short-Term Memory Neural Networks," ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 42, pp. 551-558, 2017.
[3] L. Mou, P. Ghamisi and X. X. Zhu, "Deep recurrent neural networks for hyperspectral image classification," IEEE Transactions om Geoscience and Remote Sensing, vol. 55, pp. 3639-3655, 2017.
[4] M. Russwurm and M. Koerner, "Multi-temporal land cover classification with sequential recurrent encoders," ISPRS International Journal of GeoInformation, vol. 7, no. 4, p. 129, 2018.
[5] https://www.databio.eu.

Artificial intelligence is currently penetrating many technological areas. Even though the term is used inflationary today, it often refers to machine learning, usually with deep neural networks (Deep Learning). Internet giants such as Google, Facebook, Microsoft with their almost unlimited computing capacities achieved spectacular results in image classification, text translation or in the Go game. Deep learning offers exciting possibilities in remote sensing as well. Since 2014, many quick-wins has been made for deep learning in remote sensing, yet mostly by fine-tuning existing architectures for the usual classification and detection tasks. In this work, we take particular care of the fact that EO data and problems are in many aspects different from standard imagery found in the internet, and develop tailored deep learning solutions for local climate zone classification on a global scale.

Local Climate Zones (LCZs) comprise a classification scheme for urban areas that has gained great importance across various disciplines in the last years. Originally developed by urban climatologists to describe and characterize the physical nature of cities in a way that is transferable across cultural and geographical regions, the LCZ paradigm has found its way into Earth observation as well. The 17 LCZ classes are based on climate-relevant surface properties mainly related to 3D surface structure (e.g. height and density of buildings and trees) as well as surface cover (e.g., pervious or impervious).Due to the scheme’s interdisciplinary and transferable nature, researchers have started to use the LCZ classes as a standard to describe the internal structure of urban areas. Meanwhile, we are now living in the golden Era of Earth Observation. Several German and European satellites employing novel sensor technologies and mission concepts, such as Sentinel satellite fleet and the German TanDEM-X, are already in service. These data provide geodetically highly accurate large scale geo-information from space with a dramatically ever-increasing data quantity, and serve as irreplaceable data resource for global applications.

We aim at the first ever global LCZ classification, primarily using Sentinel-1, seasonal sentinel-2 and TanDEM-X data. To enable the development and application of powerful machine learning methods, ground truth labels are manually generated by human remote sensing experts. The result of our concerted labeling efforts is called the So2Sat LCZ42 dataset, which comprises 400,673 sets of corresponding Sentinel-1 SAR, Sentinel-2 multi-spectral data and TanDEM-X SAR data, including one manually generated LCZ label per set. Moreover, for each set we have taken the independently voting from 10 experts such that the difficulty in differentiate individual classes during the human labeling process can be taken into account. Furthermore, we have developed tailored, generalizing and scalable deep learning classifiers that allow us to carry out LCZ mapping on a global scale, i.e. for all urban areas around the globe. An example deep learning architecture for LCZ classification using sentinel-2 seasonal data is shown in Figure 1. Towards an optimal spatial-spectral-temporal information exploration, we used a convolutional sub-network for high level spatial-spectral feature extraction, the outputs of which have been flown into a long short term memory (LSTM) based recurrent sub-network in order to model the seasonal behavior of the scene. The So2Sat LCZ42 is used to train and evaluate our algorithms, by separating training and testing samples according to three scenarios, namely “random splitting” (easy case), “block splitting” (realistic scenario) and “culture 10” (very challenging scenario). Classifying the 17 classes, an average accuracy of 90%, 78% and 55% for the above mentioned scenarios can be achieved, respectively, which is well beyond the state of the art. We aim at generating the first ever global LCZ map in the near future.

Figure 1. Example architecture for Sentinel-2 seasonal data: a combination of Residual Neural Network (ResNet) and Long Short Term Memory (LSTM)

Acknowledgement:
The work of is jointly supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. [ERC-2016-StG-714087], Acronym: So2Sat, www.so2sat.eu), Helmholtz Association under the framework of the Young Investigators Group ``SiPEO'' (VH-NG-1018, www.sipeo.bgu.tum.de), and the Bavarian Academy of Sciences and Humanities in the framework of Junges Kolleg.

References:
Zhu X., Tuia D.,Mou L., Xia G., Zhang L., Xu F., Fraundorfer F. (2017), Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources, IEEE Geoscience and Remote Sensing Magazine 5 (4), 8-36.

Qiu C., Schmitt M., Mou L., Ghamisi P., Zhu X. (2018): Feature Importance Analysis for Local Climate Zone Classification using a Residual Convolutional Neural Network with Multi-source Datasets, Remote Sensing, in press

Hu J., Ghamisi P., Zhu X. (2018), Feature Extraction and Selection of Sentinel-1 Dual-Pol Data for Global-Scale Local Climate Zone Classification, International Journal of Geo Information, in press.

Qiu C., Mou L., Schmitt M., Zhu X. (2018): Recurrent Residual Network for Local Climate Zone Classification with Multi-Seasonal Sentinel-2 Images, Remote Sensing of Environment, submitted.