Spaceborne Synthetic Aperture Radar (SAR) is well-suited for broad area mapping and monitoring for a range of applications including maritime surveillance, detecting new infrastructure and changes in land cover, and gathering geospatial intelligence. The availability of SAR constellations such as Sentinel-1 provide moderate to high spatial resolution imagery with a short revisit, generating rich and dense time series for data exploitation. Canada’s upcoming three satellite RADARSAT Constellation Mission (RCM) will provide enhanced operational use of SAR data for maritime surveillance, disaster management and environmental monitoring, and will further increase SAR data volumes. To better utilize the large and ever growing archives of spaceborne SAR data, there is an increasing need for effective and automated techniques to exploit the massive datasets SAR missions provide.
Artificial Intelligence (AI) and specifically deep learning (DL) have the potential to automate SAR analysis tasks, especially for mapping wide areas and for monitoring patterns of activity on a regular basis. Deep learning networks use multiple layers of nonlinear processing to learn complex patterns in the data, offering the potential for human-level performance or better on SAR exploitation tasks. Canada has emerged as a leader in AI with a strong history of contributions from researchers such as Geoffrey Hinton (University of Toronto) and Yoshua Bengio (Université de Montréal). Canada continues to be a hotspot for AI research with a growing ecosystem of contributions from government, universities, established companies (Google, Microsoft, etc.) and start-ups. As the major Canadian developer of SAR missions and provider of derived information services, MDA is keen to leverage these recent advances in AI to better exploit SAR imagery.
In this context, MDA is applying deep learning techniques to SAR and other remotely sensed data to increase automation, to increase data throughput, and to develop new applications and products. However, as is widely recognized, modelling is only a small part of an AI solution, where infrastructure to support data ingest, preparation of training data and storage of results is required for a reliable, scalable and reproducible deep learning system.
Over the last year, MDA has been defining and implementing an end-to-end deep learning pipeline to facilitate the development and use of DL techniques for remotely sensed images. Our design emphasizes the following characteristics:
• Leverages open source software with permissive commercial licenses.
• Is able to be deployed cross platform, including both on-premise and in the cloud.
• Integrates state-of-the-art deep learning frameworks with a focus on TensorFlow.
• Is computationally efficient and allows the processing of very large remotely sensed images.
• Enables analysts to focus on developing and testing deep learning models rather than on manually configuring inputs and outputs.
• Creates reproducible datasets, models and results.
Main components of the pipeline include:
• Image and ground truth ingest and pre-processing to create Analysis Ready Data. For SAR data, in particular time series of data, this may include image cropping, co-registration, speckle filtering and geocoding. We have created a Python wrapper library to allow us to chain together proprietary algorithms to perform these basic sub-tasks for RADARSAT-2 imagery, with the capability to extend to additional sensors including RCM and Sentinel-1.
• An image database (DB) to store satellite imagery and relevant metadata. The image DB leverages OpenDataCube (ODC), an open-source library for cataloguing satellite imagery. The image DB provides consistent image formats for downstream pipeline components and allows users to query across generic acquisition parameters (e.g. date range, area of interest) and sensor-specific parameters (e.g. detected or complex data for SAR, the type of pre-processing applied or desired radar polarizations).
• A labelling tool and accompanying label database to manually annotate and store labelled training data. We have extended an open-source labelling tool called CVAT (Computer Vision Annotation Tool) to support the labelling of large, geospatial time series of data. This includes breaking up large images into tiles for labelling, and auto-generating initial labels for time series of data to reduce annotation time. For SAR data, imagery is rescaled from radar backscatter with a large dynamic range to decibel-scaled 8-bit imagery that is more easily interpretable.
• Reproducible dataset creation of training image chips and corresponding labels through interaction with the image and label databases. We are working to support various task types including object detection, chip classification and semantic segmentation. Users can split data into train, validation and test sets for hyperparameter tuning and unbiased model evaluation.
• A deep learning training module that accepts records from the dataset creation module and model definitions and hyperparameters from the user to create and run experiments. Models are written using TensorFlow, a high-performance numerical library for training and inference of deep neural networks that can scale to high performance computing architectures such as GPUs.
• A deep learning database stores the outputs of each experiment including trained models, accuracy metrics, and hyperparameters. Consistent information on each experiment is collected, improving reproducibility and allowing users to perform meta-analysis on series of experiments run over a period of time or across projects.
Throughout the pipeline we are making use of a technology stack including Docker, Django, PostgreSQL and REST APIs. The labelling tool and the image, label and deep learning databases are each being designed as a service with an external facing REST API providing convenient access via HTTP requests. Applications are deployed in Docker containers to simplify deployment across multiple environments. The pre-processing, dataset creation and deep learning training modules are deployed as Python libraries that can easily be imported and called by users. The majority of the pipeline is built using Python, as this is the primary programming language used for data science, and it provides a good fit with our existing machine learning workflows.
The deep learning pipeline allows for fast prototyping and evaluation of deep learning models in the MDA R&D group. Examples of applying the pipeline for object detection (e.g. ships, aircraft, buildings) and segmentation tasks (e.g. land cover and road extraction) will be presented using both SAR and Electro-Optical data.
Going forward we will continue to add more functionality to the pipeline including support for additional sensors, new tasks (e.g. instance segmentation) and baseline deep learning architectures exploiting the spatial, spectral and temporal dimensions of remotely sensed data.
