The cost of accurately labelling enough data to train convolutional neural networks (CNNs) can easily prohibit their use in task-specific applications and services, i.e. when unsupervised domain adaptation is not possible due to a lack of existing datasets. This is especially true for the high-cost area of remote sensing (RS), where pixel-level ground truth is often not feasible or even possible. Therefore, in an operational setting ground truth is often only available in form of point labels, a single pixel that corresponds to distributed objects. This weakly supervised learning (WSL) scenario introduces a set of unique challenges, which necessitates different approaches than in previous work [Papadopoulos et al., 2017], [Zhang et al., 2015], [Han et al., 2015], and [Bilen and Vedaldi, 2016], namely extreme class imbalance and incorrectly labelled pixels around the point label. In this work, our data set has single-pixel labels each corresponding to a small but distributed ship, and we present novel approaches for applying deep CNNs to this WSL scenario. The motivation stems from the operational use-case of ship detection based on SAR images. Ship monitoring is an essential tool in the prevention and prosecution of illegal, unreported and unregulated fishing, attribution of oil spills to sources and general maritime situational awareness including tracking how large-scale shipping trends evolve. Compared to optical RS images, SAR is robust towards cloud coverage and invariant to sunlight, providing the all-weather all-night capability necessary to see the whole picture and reliably react to crises.

In our use-case, it is common to encounter images where there are only around 10 ships in a 1000x1000 pixel image. This results in an extremely unbalanced 1:100,000 positive:negative ratio on the pixel level where we observe that current CNN-based segmentation methods such as [Kampffmeyer et al., 2016] and [Lin et al., 2018] tend to fail. Further, as targets are actually spread out over several pixels, meaning that for each ship we should expect several predicted positives, instead of the single pixel (which is not necessarily perfectly placed at the ship’s center), traditional training will erroneously punish correct classification of surrounding ship pixels.

We, therefore, in this work propose a novel cost function that accounts for the extreme imbalance as well as the incorrectly labelled pixels surrounding the point labels. Our cost function introduces a region of forgiveness around each label, where the detector is not fully penalized for positive predictions. We leverage prior knowledge about the size and shape of our targets and compare different strategies for the shape and level of forgiveness of these regions and illustrate the benefit of our approach. The traditional deep learning strategy for ship detection would be to view the labels as relating to local image tiles centered on the label pixel, in order to train a CNN to differentiate between tiles that are centered around ships and those that are not. Our proposed approach avoids such tiling, instead classifying the entire image in a parallel manner. This gives the network access to a larger area of context around the ship candidate and promises speedups which are essential to making this a viable method in near real-time services.

The novel cost function and providing our network with the ability to forgive errors allows us to train a deep CNN on our weakly labelled and extremely unbalanced data. Compared to the traditional constant false alarm rate (CFAR) method, our approach allows us to look at much larger areas at a time, includes spatial context, avoids assumption on the physics of the image acquisition process by implicitly learning the background class based on the training data, and avoids the potentially demanding computation of the background distribution parameters. The presented methodology can significantly lower the effort of labeling over a wide range of detection problems and by extension the cost of introducing new services using CNNs on RS data.

References

[Bilen and Vedaldi, 2016] Bilen, H. and Vedaldi, A. (2016). Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2846–2854.
[Han et al., 2015] Han, J., Zhang, D., Cheng, G., Guo, L., and Ren, J. (2015). Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Transactions on Geoscience and Remote Sensing, 53(6):3325–3337.
[Kampffmeyer et al., 2016] Kampffmeyer, M., Salberg, A.-B., and Jenssen, R. (2016). Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–9.
[Lin et al., 2018] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Doll´ar, P. (2018). Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[Papadopoulos et al., 2017] Papadopoulos, D. P., Uijlings, J. R., Keller, F., and Ferrari, V. (2017). Training object class detectors with click supervision. arXiv preprint arXiv:1704.06189.
[Zhang et al., 2015] Zhang, D., Han, J., Cheng, G., Liu, Z., Bu, S., and Guo, L. (2015). Weakly supervised learning for target detection in remote sensing images. IEEE Geoscience and Remote Sensing Letters, 12(4):701–705.

Although convolutional neural networks (CNNs) have shown a remarkable performance in semantic segmentation problem [1], the major limitation is their assumption that the training data are static, which is almost never the case in real-world applications, as new data are collected everyday from all around the world. Besides, the label maps obtained from different sources may have separate classes. In addition, storing massive amount of satellite images is not always possible. In this context, it is of paramount importance to propose a continual learning system that can learn new classes on new training data without hindering segmentation capabilities for the previous classes, even if the entire previous data are not stored. Let us explain a real-world problem, where we are provided satellite images from Austria and the label maps for building and tree classes in the beginning. We then retrieve new images from Germany, having the ground-truth for water class. Finally, we are given images collected from France and annotated for road and railway classes. We assume that every time when new data are retrieved, only a small portion of the previous data is stored. In such an experimental setup, the recent approaches suffer from catastrophic forgetting: an abrupt performance degradation for the previous classes when new classes are added to the network on the data, having no annotations for the previous classes [2]. We propose an approach that enables to learn new classes on the new data, while retaining performance for the old classes. When a new test image comes, the network is able to segment the image for all the presented classes. The real-world continual learning problem described here is illustrated in the figure below.

Figure 1: An example of real-world continual learning problem

Our network architecture is a variant of U-net [3]. In the final layer of our network architecture, there are as many filters as classes, which are responsible for generating a binary predicted map for each class. The main points of our approach is adaptation, where the network learns new as well previous classes on the new data, and remembering, where previous classes are reminded to the network on the previous data. For adaptation, we keep a frozen copy of the previously trained network, which we use as a memory for the updated network in absence of annotations for the previous classes. The first loss function used in adaptation is classification loss Lclass which quantifies the mis-classification rate between outputs for the new classes from the updated network and the new ground-truth. The second one is distillation loss Ldistil that measures disparity between outputs for the previous classes from the memory and the updated networks. The overall loss that is optimized during adaptation is defined as:
Ladapt = Lclass + Ldistil
When we compute Ldistil, we use the soft probability map generated by the memory network instead of the hard classification map to reduce the error rate in the prediction. For remembering, we systematically feed samples from stored, small fraction of the previous training data. We optimize Lrem that is the mis-classification rate between outputs for the previous classes from the updated network on the old data and the previous ground-truth. How often and on which previous data Lrem would be optimized need to be determined by the user.

We conduct experiments on the data collected from 22 different cities, out of which 11 are located in France, and 11 in Austria. We are provided full annotations for building, tree, road, railway and water classes. We use 4 cities as test, the other 18 as training. We split the training images into three groups. We assume that the images in the first group have label maps for building and tree classes, the ones in the second group have annotations for road and railway classes, and the images in the third group have target maps for water class. We suppose that the training data are streamed in this order: group1, group2, group3. Our experimental results prove that our approach can learn building and tree from group1, road and railway in group2, and water in group3. Every time when new classes are added from a new group, performance for the classes in the previous groups is preserved. All the classes can be detected in test images successfully.

More details on the proposed methodology can be found in [4].


ACKNOWLEDGMENT

The authors thank ACRI-ST and CNES for initializing and funding this study. The authors also would like to thank Luxcarta for providing the satellite images and their annotations.

REFERENCES


[1] Maggiori, E., Tarabalka, Y., Charpiat, G., and Alliez, P., “High-resolution semantic labeling with convolutional neural networks,” IEEE TGRS 55/12 (2017).

[2] Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y., “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv (2013).

[3] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” MICCAI , 234–241, Springer (2015).

[4] Tasar, O., Tarabalka, Y., and Alliez, P., “Incremental learning for semantic segmentation of large-scale remotesensing data,” arXiv (2018).

When looking at the Earth's surface from above using remote sensing, be it with an imaging satellite or an airborne camera, the apparent absolute orientation of the objects in the image does not depend on the nature of those objects, but rather on the pose of the sensor. However, this pose is irrelevant for tasks such as scene classification (the assignment of a single semantic class label to an image tile) or semantic labelling (the assignment of a class to each pixel of the image). This is in contrast to what one can observe when considering ground-based photography, where both the camera and the objects imaged are aligned with the vertical direction defined by the Earth's gravity, therefore giving rise to a preferred orientation. For this reason, the absolute orientation of an object in a ground-based image might be relevant to understand the scene, while in the case of remote sensing images, which can be considered isotropic for most purposes, the absolute orientation of the objects should have no effect on the final interpretation.

Motivated by the above, it becomes desirable to impose certain simple behaviours with respect to the rotation of the input image in problems involving remote sensing imagery. Two main problems can be highlighted: (1) rotation invariance in scene classification, meaning that the output should remain unchanged when the input image is rotated; (2) rotation equivariance in semantic labelling, meaning that the output map is expected to rotate according to the rotation of the input.

In this paper we consider the inclusion of prior knowledge about orientation in a Convolutional Neural Network (CNN). We propose a modified convolution operator that can be used in any CNN architecture. In details, the convolution operator is first applied using rotated versions of the learned filters and both the maximal activations across orientations and their angle are recorded. These two values are stored as a vector field that encodes information about magnitude and rotation. By doing so, one can inject any desired behaviour with respect to the rotation of the input. Our method allows extracting each feature, at each layer of the CNN, at multiple orientations and then keep only information about the maximum feature activation and the orientation that caused it. For this reason, the representation learned is very compact.

We study the behaviour of our proposed RotEqNet model on two datasets aiming at precise semantic segmentation in urban environments at subdecimeter resolution. On one hand and in both cases, accounting for rotation equivariance allows to reduce the size of the CNN models (in terms of the number of learnable parameters involved) by an order of magnitude and without any loss is terms of accuracy. This opens opportunities in training CNN models with smaller labelled dataset, as prior knowledge is used to reduce the amount of data structure that needs to be learned directly from the data.
On the other hand, our proposed RotEqNet performs equivalently to competing specific CNN architectures encoding equivariance to rotation, thus showing that our compact representation pushing only maximal orientation activations through the network keeps enough information for the task.

We propose a new method to count objects of specific categories that are significantly smaller than the ground sampling distance (GSD) of a satellite image. This task is hard due to the cluttered nature of scenes where different object categories occur. Target objects can be partially occluded, vary in appearance within the same class and look alike to different categories. Sentinel-2 satellite configuration provides since 2015 multi-spectral images of up to 10 meters GSD.

However, for applications that need more high-resolution evidence like detecting and counting objects (e.g, supply chain management, financial industry), spatial resolution is too poor to apply traditional object detectors like Faster R-CNN (Ren et al., 2017). In this work, we thus propose to circumnavigate explicit object detection by turning the counting problem into a density estimation task. Furthermore, we add semantic segmentation to be able to count objects of very specific object categories embedded in cluttered background.

Semantic Segmentation is a standard task in computer vision and has seen significant performance gains since the comeback of deep learning. The major goal is predicting a class label per pixel over an entire image. Typical objects classes of interest are buildings, persons and vehicles that are (i) clearly visible in the image, (ii) large in size (usually several hundreds of pixels) (iii) and can be distinguished from background and further classes primarily relying on shape and RGB texture.

Objects like cars, trees, and buildings constitute a single pixel or less in remote sensing imagery of Sentinel-2 (10m GSD) and Landsat (15m GSD). While this resolution is sufficient for semantic segmentation of large, homogeneous regions like crops, counting individual object instances in cluttered background becomes hard. Here, high spectral resolution comes to the rescue. Deep learning techniques can greatly benefit from high spectral resolution that conveys object information invisible to human sight. It can learn complex relations between spectral bands to identify object-specific spectral signatures that support pixel-accurate semantic segmentation.

To distinguish objects of different classes, our approach combines density estimation with semantic segmentation in an end-to-end learnable convolutional neural network (CNN) to count objects of down to 1/10 the size of the GSD. We compare our proposed architecture with state-of-the-art semantic segmentation methods for terrestrial images that use among other ideas atrous convolutions to prevent lowering the resolution of the learned features keeping a large receptive field (Chen et al., 2017).

It is impossible to manually segment individual trees or cars in the Sentinel-2 images of 10 meters GSD. We thus resort to Google Maps overhead images of much higher resolution for groundtruth labeling. We apply the Faster R-CNN Object detector to very high-resolution (1m GSD) Google Maps images to identify and count reference objects. The detector is tuned to achieve high recall to then manually remove false positive predictions. This allows us to obtain a highly detailed count per area that is used as ground truth to test our model on lower resolution satellite imagery.

Our approach consists of using ResNet blocks, which contain mainly convolutional layers, to obtain features from the input image without any downsampling. Then, for each task, we add an independent convolutional layer at the end of each architecture. For all training experiments, our Loss is defined as the sum of (1) cross entropy of the estimated and reference class (object of interest or backround) and (2) the euclidian distance of the estimated and reference count.

In order to test how robust deep semantic density estimation is to changes in texture, object density and size, we create four different datasets. Three datasets contain trees (olive, coconut, oil palm) with different planting patterns and one dataset contains cars. The tree and car sizes with respect to the pixel area size were approximately of 1/3 and 1/10.

Experiments on four different objects show that deep semantic density estimation can robustly count objects of various classes at sub-pixel scales in cluttered scenes. In our tests, we were able to predict the total object count with an error of less than 5%. For the semantic segmentation task of Olive Trees, our architecture obtained Intersection over Union of 0.86 and precision of 0.90 in our test set; in contrast to other architectures that showed poor performance. See Figure 1 for a visualization or our results. Experiments with trees (Olives, Coconuts and Palms) show the importance of infrared bands in the prediction. In contrast, Cars benefited mostly from the high spatial resolution of the RGB bands.

We have proposed end-to-end learnable deep semantic density estimation for counting object instances of fine-grained classes in cluttered background. Results show that counting objects of sub-pixel size is possible for 10m GSD satellite images. Experimental evaluation with four datasets shows that our method is robust to change in object types, background, and object density. It turns out that a shallow network specifically designed for satellite imagery of 10m GSD and sub-pixel objects outperforms more sophisticated, state-of-the-art architectures from computer vision. This signifies that direct application of networks tailored for vision to remote sensing images should be done with care. In our experiments, we find that any down-sampling operation inside the network risks loosing precious details. We should thus always keep in mind the particularities of remote sensing imagery in terms of object scale, GSD, (nadir) perspective and (high) spectral resolution. If carefully considered during the network design process, these specific properties offer new possibilities in network design.



References

Chen, Liang-Chieh, et al. "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs." IEEE transactions on pattern analysis and machine intelligence 40.4 (2018): 834-848.

Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances in neural information processing systems. 2015.

Change detection in VHR images is important for several applications including urban planning, disaster management, and cadastral map updating. In the last few years, a new generation of Very High spatial Resolution (VHR) satellite sensors have been launched, which can acquire images having spatial resolution of one meter or less. The availability of VHR data allows the analysis of urban areas at high spatial details, making it possible to analyze single man-made structures, e.g., buildings. In this context, several techniques for the analysis of urban areas have been developed exploiting both passive (optical) and active (SAR) sensors [1, 2]. SAR sensors are particularly useful in the applications that require quick response (e.g., disaster management) as they effectively map the affected areas irrespective of the time of the day or the weather conditions. Currently, several satellite with SAR sensors are operating (e.g., TerraSAR-X, Tandem-X, COSMO-SkyMed constellation) that can acquire VHR images. However, the increased resolution introduces new challenges to analyze as the need to deal with the high spatial correlation among pixels. In VHR SAR images, objects like buildings exhibit inhomogeneous signature due to the scattering contributions from the primitive composing them. Moreover, side looking illumination required by SAR systems causes shadows and distortions. Spatial context needs to be modeled to detect changes in such images by properly taking into account the high geometrical details of VHR data. In the literature, works exist dedicated to unsupervised building CD that exploit shallow features to capture the spatial context [1, 3]. Such methods have limited capability in capturing object level information. Recently, Convolutional Neural Networks (CNNs) have made considerable progress in object recognition and image processing. CNN-based features, also known as deep features, effectively capture high level semantically rich visual features. Training a CNN-based algorithm requires a large amount of labeled training data that are not available in the context of SAR multi-temporal analysis. To overcome this, we exploit the availability of SARptical dataset [4] that provides large amount of coregistered TerraSAR-X VHR images and aerial optical images. By training for the sub-optimal task of transcoding SAR images into optical images, a multi-layered CNN based Generative Adversarial Network (GAN) learns relevant visual features from SAR images [5]. We employ this CNN as a feature extractor from multi-temporal SAR images. Coregistered pre-change and post-change VHR SAR images are converted to dB scale and processed through the multi-layered CNN to obtain deep features. Deep features implicitly model the spatial context information of each pixel and reasonably capture object level information in SAR images. Following this, an automatic and unsupervised variance-based feature selection strategy is designed that layer-wise selects change-relevant deep features. Selected deep features from multiple layers of the CNN enable us to obtain deep feature hypervectors that are pixelwise compared to obtain deep change hypervectors. Assuming that unchanged pixels yield similar deep feature hypervectors, whereas changed ones do not, deep change magnitude is calculated from deep change hypervectors and analyzed to distinguish changed pixels from unchanged pixels. Deep change hypervectors corresponding to changed pixels are binarized and further analyzed to identify different kinds of change. Experiments performed on a dataset made up of a pair of X-band VHR SAR images acquired by COSMO-SkyMed constellation on the city of L'Aquila (Italy) confirm the effectiveness of the proposed approach.

References
[1] Marin, C., Bovolo, F. and Bruzzone, L., 2015. Building change detection in multitemporal very high resolution SAR images. IEEE Transactions on Geoscience and Remote Sensing, 53(5), pp.2664-2682.
[2] Saha, S., Bovolo, F. and Bruzzone, L., 2018. Unsupervised multiple-change detection in VHR optical images using deep features. In IEEE Geoscience and Remote Sensing Symposium (IGARSS ‘18).
[3] Brett, P.T. and Guida, R., 2013. Earthquake Damage Detection in Urban Areas Using Curvilinear Features. IEEE Trans. Geoscience and Remote Sensing, 51(9), pp.4877-4884.
[4] Wang, Y. and Zhu, X.X., 2018. The SARptical Dataset for Joint Analysis of SAR and Optical Image in Dense Urban Area. arXiv preprint arXiv:1801.07532.
[5] Ley, A., Dhondt, O., Valade, S., Haensch, R. and Hellwich, O., 2018, June. Exploiting GAN-based SAR to optical image transcoding for improved classification via deep learning. In EUSAR 2018; 12th European Conference on Synthetic Aperture Radar (pp. 1-6). VDE.

Data augmentation is a well known technique that is frequently used in deep learning tasks to increase the number of training instances and hence decrease model overfitting. It is widely used in classification tasks as it helps the model to robustly classify objects placed in different orientation, size, illumination and viewing points. To mitigate this, simple transformations can be applied to the training datasets such as translating, scaling, flipping and rotating of training samples. While data augmentation techniques are of vital importance for classification and object recognition tasks, the effects of employing it for super-resolution tasks are not well identified. Moreover the peculiarity of satellite data features are not covered yet in the super-resolution deep learning domain.
In this abstract we propose a data augmentation technique that can further boost the performance of satellite image super resolution tasks. A super-resolution convolutional neural network (SRCNN) was adopted as a state-of-the-art deep learning model to test the proposed data augmentation technique. To train the model, pairs of low-and-high resolution training samples are generated to teach the model how to map a low resolution instance to a high resolution vector space. The low resolution version is synthesized by downscaling the original instance while the original resolution instance is fed to the model as a target high resolution output. Both Sentinel-2 and Landsat-8 datasets were used to train the model and test the augmentation effects on different resolution data (10 meters for Sentinel and 30 meters for Landsat-8).
Different augmentation techniques were studied to investigate their relative importance and accuracy gains. We categorized the techniques into instance based and channel based augmentation. The former refers to the standard approach of creating new data instances through applying simple image transformations to the original images such as rotations, flipping and translations, while in the latter we add auxiliary channels (or custom bands) to each training instance which help the model learn useful representations. The effects of the proposed channel augmentation approach were evaluated qualitatively and quantitatively. A quantitative evaluation was carried out using Peak-Signal-to-Noise-Ratio (PSNR) metric and the achieved score of the proposed method exceeded the traditional instance-based data augmentation approach. The qualitative evaluation was applied using an object-based classification task where the classification of objects in the super-resolved instances against the original images was compared. The original image was considered as the high resolution reference to evaluate the object classification accuracy. The experimental results revealed that our proposed channel augmentation methodology was able to present useful insights to the model without damaging object texture and formation.