One the one hand, widespread violations of environmental and human rights, to a point verging on existential risk. On the other, more abundant earth observation data than ever, and powerful Machine Learning “Artificial Intelligence” algorithms whose capabilities seemed science-fiction just a few years ago. We are bringing the two together to help domain experts scale their efforts, actions, and impact beyond what used to be possible. From our early use of Deep Learning and crowdsourcing to detect destroyed villages in remote conflict zones for Amnesty International, to our recent multiframe super-resolution algorithms to open affordable agricultural and limnology monitoring, we show our progress towards our dream of an AI-driven platform enabling multipartite action at the scale of a planet.

Unibap introduced x86 instruction based heterogeneous computing for space applications with increased reliability features in 2015[1]. Motivations for the new architecture was the ability to share a common software architecture between industrial automation, robotics, and artificial intelligence and space applications, thus, enabling the use of advanced frameworks in radiation environments at minimal changes (e.g. Tensorflow, PlaidML, Robotic Operating System (ROS), Matlab/Simulink, Linux distributions, Virtual machines, OpenCV, OpenCL, Caffe2, Docker containers). Furthermore, open source and partially closed source software can be monitored using a SafetyChip companion module to provide a hard-soft monitoring solution for increased radiation tolerance to protect the execution distributed over GPU, CPU, FPGA or other attached processing accelerators (e.g. VPU, TPU) on PCIexpress, USB, SATA or FPGA bridges.

Unibap will discuss the common industrial-space software platform based on the Deep Delphi iX5 (DD-iX5) onboard data processing solution. The DD-iX5 provides x86 heterogeneous computing together with a plethora of input/output and a stacking interface for flexible expansion, including fast local solid-state storage (up to 1 TB) in a CubeSat compatible form factor. Typical additions to the standard architecture with multi-core CPU, 128 shader GPU, micro-processor and FPGA are different kind of artificial intelligence accelerators (e.g. Intel Movidius Myriad X VPU or embedded MXM GPUs).
The heterogeneous nature of the architecture is key for offering a high degree of software flexibility on-orbit to sustain business models that allow a high mix of different tasks to sustain multi-sensor, multi-user, and multi-mission objectives effectively.

The heterogeneous compute architecture is demonstrated with various benchmarks to illustrate the flexibility and agility of the onboard data processing solution, and planned roadmap for intelligent processors on 14 nm, 12 nm, and 7 nm.

References:
[1] F. Bruhn, K. Brunberg, J. Hines, L. Asplund and M. Norgren, "Introducing radiation tolerant heterogeneous computers for small satellites," 2015 IEEE Aerospace Conference, Big Sky, MT, 2015, pp. 1-10. doi: 10.1109/AERO.2015.7119158§

Planet operates the largest constellation of Earth-observing satellites in human history, collecting high-resolution 3-5m imagery of the entire Earth’s landmass every day. The novel cadence and scale of the observations is unlocking unprecedented opportunities for tracking and discovery of natural or man-made phenomenologies which would otherwise escape detection. Intersecting dense time series of global observations with modern deep learning solutions allows us to establish a robust spatiotemporal baseline against which to measure change and take a daily pulse of the planet.

Our focus has been in training and optimizing of deep learning architectures on this new type of data to develop a new family of monitoring solutions. Man-made structures such as roads, buildings, and some classes of transient objects are among the information layers that we are beginning to extract from our imagery reliably and at scale. These offer new opportunities for timely augmentation of OpenStreetMap in disaster management situations affecting developing countries, especially in helping update road networks, identifying the extent of the damage caused by a natural disaster and facilitating the work of relief crews on the ground. The high cadence of these spatial analytic feeds increases our confidence in tracking permanent change associated with urbanization and improves our knowledge of how human settlements grow. For instance, we will demonstrate how our automated road and building extractions over land parcels can serve as a proxy for several of the urban categories in the European Urban Atlas and help automate its maintenance by highlighting change at an earlier stage, possibly providing a mechanism for monthly updates in the future. The combination of improved remote sensing and advanced analytics enables us to cost-effectively map the fine structure of the biomass stored in tropical forests and measure carbon emissions at national scale, high resolution and high frequency. Large scale spatiotemporal mapping of transient objects on a large scale offers even more intriguing possibilities. The temporal depth of our ongoing collection coupled with nearly two years of global daily coverage provides enough data for advanced pattern classification and discovery. We will show patterns associated with daily marine traffic over time windows of several months to a year in the proximity of large harbors, in the open sea, nearby shipping lines, and demonstrate potential departures from a “spatiotemporal norm”.

A longer-term R&D goal in our analytics road map may be the development of a general-purpose scalable architecture for change detection, with the aim of producing daily change heat maps to drive awareness of change, enable information discovery and, also, perform area reduction for more specialized extraction of thematic information layers.

European space agencies and Industry rely on AI to develop new EO capabilities, integrating intelligence on-board. Thus, this paper presents different solutions useful for simplifying deep learning neural networks (DNN) to be used for data analysis and information extraction on-board. Having DL on-board will provide data and downlink reduction, allow acquisitions adaptation (pointing), inter-mission satellite reactivity and speed up useful information availability. Nevertheless, existing state of the art on DNN which really performs well on EO image analysis are huge networks requiring a lot of computation power to be run. These requirements are incompatible with power and resources limitations on-board. Even if only inference step is computed on board, the best DNN cannot be used for image analysis on on-board HW. Smaller networks have to be defined while managing the losses in reliability and efficiency. Different steps and methods exist to simplify/compress DNN including quantization, pruning, but also distillation. This last extraction step shall not bring significant loss in terms of precision and current implementations in other domains reveal gain factor of 1000 in complexity. Moreover, distillation is complementary of existing techniques to reduce DNN memory footprints (ex. EUCLID mission), compatible with precision reduction techniques for inference (N2D2) and can reuse generic VHDL code generation for running DNN on Soc FPGA HW (ex. N2D2). Other strategies for avoiding losses in results reliability, can be applied at training step by re-training. These different simplification techniques will be presented, reviewed and discussed as well as those to be performed at training stage. Some solutions of intelligent payload including DL-based analysis are also considered as well as other processing solutions including FPGA implementations of DL performed in EO and not EO domains. The most suitable DNN for object extraction on EO images are simplified with the most promising reviewed techniques for on-board processing. A performance evaluation is done and analyzed against the network complexity. The results obtained in network simplification applied at learning level and directly on DNN models will be presented.

Specific technological innovations are required to accomplish more ambitious commercial and scientific goals for Earth Observation missions. One of the key areas of potential innovation is mission autonomy: an increased degree of on-board autonomy helps in implementing more effective mission operations. In particular, functionalities like advanced event detection and feature extraction, can be implemented on-board with satisfactory results only if dedicated computing resources are available: FPGAs, GPUs and VPUs are the leading candidate for providing the required computational power. This paper presents the recent applications developed at AIKO, specifically targeted to the use on-board satellites. The paper presents three applications related to observation: cloud segmentation, maritime vehicle detection and ground target tracking, implemented and deployed to the new Intel Myriad X VPU. A comparison between the performances of the algorithms is provided, in the two cases of execution on a COTS processor and on the Intel VPU. The algorithms presented exploit state of the art Deep Learning architectures, based on Convolutional Neural Networks. Moreover, a comparison between model characteristics developed on standard Tensorflow framework is described, with respect to the OpenVINO standard required for porting the algortihms to the Myriad VPU.
The combination of state of the art Deep Learning algorithms, and innovative embedded processing units (such as the Intel Myriad), provides a powerful setup to enhance the services provided by satellites systems, making high level information from payload data available on the spacecraft.
In an era where Artificial Intelligence is making cars and other systems smarter and more autonomous, AIKO envisions the development of Deep Learning applications on-board satellites to enhance the autonomy capabilities of satellites, in order to enable faster reaction to events, more advanced concepts of operations (e.g. ground target tracking), improvement of the relevance of downlinked data, and finally enabling innovative and more effective services to ground (e.g. surveillance applications).

The vastly increased availability of high resolution Copernicus Sentinel-1 (S1) and -2
(S2) data streams requires complementary high quality in-situ data. To boost
collection efficiency novel ways to generate such high quality reference data are needed for cross-referencing, validation, and ultimately, development of robust inference
methods (e.g. machine learning) for use in near-instantaneous information retrieval
for wide-area applications. This is particularly relevant in the agricultural domain, for
which S1- and S2-revisits closely match the dynamics in the soil-canopy system
underlying the electromagnetic signal formation. In-situ data collection is a tedious
and expensive task and is often limited to small scale, and specific purpose,
scientific validation experiments. Only very limited amounts of standardized
reference data end up in the open data domain, notwithstanding large efforts like
those of the Group of Earth Observations (GEO) Joint Experiment of Crop
Assessment and Monitoring (JECAM).
The use of opportunistic street-level imagery (SLI) is well-known for geo-visualization
approaches (e.g. Google Street View, Mapillary). Deriving quantitative information
from such imagery, e.g. for characterization of crops, is less developed, but
recognised to have a significant potential [1]. The idea to “guide” SLI acquisition by
integrating existing geospatial data (e.g. for routing) and EO data (e.g. to visit
pre-marked anomalies) is relatively new [2]. Such guided SLI surveying can result in
an increased data collection efficiency compared to conventional surveys by a factor
of 7 to 8. Currently, the approach still relies on a human visual interpretation step to
tag and classify the collected SLI with the observed crop type and crop phenological
stage. The use of computer vision and machine learning algorithms to derive crop
parameters has considerable potential to further increase the efficiency of the use of
SLI in generating relevant reference data.
We demonstrate this concept with the use of SLI collected with side-looking action
cameras for a fixed route survey in Flevoland (NL) on 8 dates in the growing season
(March-October) in 2018 [3]. This survey was designed to ensure a statistically
meaningful sample density for the main crop types and their phenological dynamics
in the area and includes over 200 revisited reference parcel locations for which
detailed observations were made. The SLI was collected at 1 frame-per-second
(FPS) along the entire route and has resulted in a data set of nearly 200,000 full HD
resolution images.
We deploy machine learning using TensorFlow with a well-known low latency,
high-accuracy image recognition model (MobileNet v2), with the application of
inductive transfer learning for convolutional neural network modules. We
demonstrate the wide-scale applicability of these techniques with the use of models
that are initially trained on the large ImageNet dataset which then have their top
layers retrained in order to recognize at first instance the different crop types and
secondly the respective phenological development (BBCH) stage. Training data of
300 images per class is pre-selected based on logical filters, derived with the aid of
well known parcel geometries, in order to remove unwanted training examples
including images showing more than one crop (images at borders between parcels)
or images showing no crop at all (trash class images). Additional post-processing on
labelled data in terms of applying probability thresholds on top 1 class is leveraged
so as to eliminate instances at the lower end of the probability distribution. This
results in a significant reduction of incorrectly labelled examples, thereby increasing
overall accuracy by 30%, while at the same time keeping the number of images
significantly high. Results are presented in terms of accuracy on the individual
picture level and on an aggregated parcel level, based on a majority vote. We also
show the use of a smart GUI to rapidly quality control geotagged results and inspect
special cases.
The use of the machine learning results as validation data for Sentinel time series
will be illustrated and compared to traditional reference data. We also discuss future
perspectives, including instantaneous crop recognition using our trained models and
further integration into standardized data collection protocols, e.g. those serving CAP
“checks by monitoring” requirements.
[1] d'Andrimont, R.; Iordanov, M.; Lemoine, G.; Yoong, J.; Nikel, K.; van der Velde, M.
Crowdsourced Street-Level Imagery as a Potential Source of In-Situ Data for Crop Monitoring. Land
2018 , 7 , 127.
[2] d’Andrimont, R.; Lemoine, G.; van der Velde, M. Targeted Grassland Monitoring at Parcel Level
Using Sentinels, Street-Level Images and Field Observations. Remote Sens. 2018 , 10 , 1300.
[3] de Vries, M., van der Voet, P., van der Velde, M., Lemoine, G., d’Andrimont, R., Efficient crop
phenology surveying for EO validation using guided street-level imagery, Abstract submitted to the
Living Planet Symposium, 2019