The NATO lead Trident Juncture 18 military exercise was arranged in Norway in October and November 2018. It is the largest exercise since the Cold War. 50 000 participants, 250 aircraft and 65 vessels from more than 30 nations were part of the exercise [1]. The exercise made it possible to train realistically in both air, sea and land domains at the same time. The weather in Norway gives the NATO troops additional challenges and it is possible for them to train under extreme conditions. The bad weather and low light conditions make SAR an excellent choice to monitor areas on land and sea. SAR works in day and night and can see through clouds.

Satellite images have been ordered over the exercise areas over a period of 10 weeks. 822 satellite images have been collected. Sentinel-1A, Sentinel-1B, Sentinel-2, TerraSAR-x, Cosmo SkyMed have been ordered and collected over land and sea. Some of the images were collected before the exercise, but most of the images were ordered during the exercise. The Sentinel-1A and -1B images were downloaded directly from the satellite to ground through the national Norwegian ground segment at Kongsberg Satellite Services in Tromsø, Norway.

The paper will present an overview over the images acquired before and during the exercise, the modes of the images and the coverage areas of the satellite images. Maps will be used to show the coverage and the prioritized areas. Satellite Tool Kit will be used to show the possible coverage during the same periods as the acquired images. How the different SAR satellites can be used together is analyzed and the gap due to the sun synchronous orbits used will be taken into account. How Artificial Intelligence has been used to analyze the large amount of data is explained. We have an automatic ship detector, Aegir, as well as Nevron (machine learning) and a Multi Hypothesis Tracker (to combine all the information) at FFI to help analyze the data and to get an overview of the current situation.

The orbits and coverage of the different satellites can be analyzed before the satellites pass over the areas (Space Situational Awareness). That way military knows where they can be observed and where they can safely move without being observed.

[1] https://forsvaret.no/en/exercise-and-operations/exercises/nato-exercise-2018

In the field of Earth observation (EO), data fusion of satellite data is quickly gaining momentum; however, data fusion processing techniques, in particular data association techniques, are still far from being optimised. This is partly due to the intrinsic complexity and diversity of the different EO datasets and the temporal gap between corresponding acquisitions. This gap often invalidates the attempt of fusion for specific applications in which parameters to be monitored are highly sensitive to time variations.

In some specific applications, such as those in maritime surveillance, and for specific datasets, such as the Synthetic Aperture Radar (SAR) and Automatic Identification System (AIS) datasets from the first UK spaceborne SAR mission, NovaSAR-1, these problems are mitigated. With both SAR and AIS payloads co-located on the NovaSAR-1 platform, a near-contemporaneous acquisition of the two systems is possible. This innovative concept offers a near real-time maritime surveillance system, capable of providing a global picture of human activities at sea (see Fig. 1a).

However, an additional problem that is unique to SAR and AIS datasets is the data association in high-density shipping areas (where correspondences are also unknown a priori). Ships detected in SAR imagery can be wrongly associated with AIS reports in congested areas (e.g. ports and busy shipping lanes) and result in an erroneous and/or inaccurate picture.

In this research, the best techniques of fusion for near-contemporaneous SAR and AIS datasets from NovaSAR-1 are studied. Specifically, data association techniques are quantitatively compared by performing a series of Monte Carlo tests. The problem of SAR and AIS data association is first described in terms of a 2-D assignment problem [1], where a global nearest neighbour (GNN) algorithm is used to provide an optimal solution to the assignment problem, and then is extended to establish an m-best multidimensional assignment technique. Next, the data association is formulated as a point set registration problem [2], which uses the RANSAC algorithm to return a 2-D geometric transformation that optimally aligns SAR ship detections to AIS-reported positions.

The Monte Carlo tests are performed using a real satellite-based AIS dataset acquired from the English Channel on 07 June 2016, and SAR ship detections are simulated by reckoning the AIS dataset forward in time along a geodesic on a WGS84 reference ellipsoid. By reckoning the AIS dataset forward in time, the performance of the algorithms can be measured as a function of the temporal difference (or gap) between the sensors' acquisition time (see Fig. 1b and Fig. 1c). To produce the random element in the Monte Carlo tests, the speed and heading for each ship are changed every iteration according to the statistical distributions derived by the AIS dataset.

The results show that assignment-based techniques (GNN) offer the best performance, both in terms of association accuracy and computational performance. In particular, the m-best multidimensional assignment technique outperforms conventional techniques found in the literature (e.g. k-NN) on average by 10% in terms of association accuracy (see Fig. 1b) and baseline GNN algorithms are over 40 times faster in execution time (see Fig. 1c). By using a multidimensional approach, i.e., using auxiliary information such as ships’ heading, dimensions, etc. a greater level of correspondence of the data is achieved and non-cooperative or “rogue” shipping can be better identified. The proposed data association technique is consistent with being used in an operational environment, especially in high-density shipping areas, and a case study with real datasets will be presented at the time of the conference, following the completion of the commissioning phase of NovaSAR-1.

References:

[1] Y. Bar-Shalom and W. D. Blair, “Multitarget-Multisensor Tracking: Applications and Advances (Vol. 3)”, ser. Artech House Radar Library. Artech House, 2000, ISBN: 1580530915.

[2] M. A. Fischler and R. C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981. DOI: 10.1145/358669.358692.

Earth observation (EO) data delivered by remote sensing satellites provide a basic service to society, with great benefits to the civilian. The data is nowadays ubiquitously used throughout society for a range of diverse applications, such as environment and resource monitoring, emergency management and civilian security.
Over the past 50 years, the EO data chain that has been mastered involves acquisition process of sensor data on-board the satellite, its compression and storage on-board, and its transfer to ground by a variety of communication means, for later processing on ground and the generation of the downstream EO image products.
The EO-ALERT project, an H2020 European Union research activity, addresses the challenge of a “high speed data chain” and the need for increased EO data chain throughput. EO-ALERT proposes the definition and development of the next-generation EO data and processing chain, based on a novel flight segment architecture that moves optimised key EO data processing elements from the ground segment to on-board the satellite. The objective is to deliver the EO products to the end user with very low latency for increased throughput.
This paper presents the activities performed during the first part of the EO-ALERT project, focussing on the definition of the user requirements for the EO-ALERT EO data processing chain, based on the identified market needs and application scenarios, now and in the future. Several application scenarios for EO data are reviewed and those which need and drive a high speed data processing chain are identified, considering commercial and institutional applications and focusing on those applications with a low latency need.
Considering the innovative on-board processing chains implemented in the EO-ALERT project, the end users consider that the corresponding technology advances can enable very competitive and effective applications in the maritime surveillance and the disaster management, especially to cope with extreme weather events. These applications require a high responsiveness to events, reducing the response time to few hours, or even to minutes, after an emergency situation arises.
Ship detection application includes, among others, traffic surveillance, protection against illegal fisheries, oil discharge control and sea pollution monitoring. One of the main objectives in maritime surveillance in Europe is to improve the European capability in the field of ship detection in both open and coastal waters for the purposes of security, providing observations on very short notice and near real-time detection. Flexibility and responsiveness for the provision of EO data are very important for maritime surveillance and security services. Two European EO missions have been selected to develop and test the on-board data processing chain for the ship detection scenario: the TerraSAR-X satellite mission embarking an X-band Synthetic Aperture Radar instrument able to image day and night under almost all meteorological conditions; and the Very High Resolution and agile optical satellite, Deimos-2 for detection and classification of small ships.
There are many different segments of society that benefit from an early detection and warnings of the extreme weather events. Defence, police and emergency authorities use extreme weather warnings to reduce loss of human life and property by providing people with instructions, building containers for foods, evacuating people from risky areas, just to cite a few. Also, environmental and road authorities need these kinds of warnings to reduce loss of natural resources and human life. Aviation needs information about storm occurrence and exact position in order to plan flight routes to reduce loss of human life, property and fuel. The shipping and fishing trades take advantage of this information to re-plan and take less risky routes. The industry energy sector uses wind speed warnings, associated to convective storms, to reduce windmill breakdown.
Two meteorological phenomena are going to be detected in the extreme weather scenario: convective storms and surface winds overseas and oceans. The main objective is the provision of early warnings for these two phenomena from satellite information. Considering the specific needs for the meteorological scenario, the following current sensors have been considered: SEVIRI on board Meteosat Second Generation to be used operationally, the radar on board TerraSAR-X to be used operationally and SLIM6 on board Deimos-1 to be used for research purposes.
A high-level mission analysis for the above mentioned assets is presented, encompassing: coverage analysis, ground station contact analysis and data latency. For ship detection, a scenario that combines optical images from Deimos-2 and SAR images from TerraSAR-X is studied, providing information on possible common observation windows.
An experimental campaign at the end of the project is intended to validate the technology performances in the experiment scenarios. The objective is to acquire the necessary in-situ data at the same time that SAR and optical remote sensing images are acquired on the region of interest, so as to provide a holistic knowledge of the monitored area and data for validation of the processing chain performances.

In the framework of its Research, Technology Development and Innovation (RTDI) activities, the European Union Satellite Centre (SatCen) is assessing how state-of-the-art technologies and innovative geospatial data management solutions can improve its operational capabilities as well as the offering of EO products and services to Space and Security stakeholders.

The free, full and open access to EO data provided by the Copernicus Programme has been a game changer, creating the premises for new and innovative services in different domains. The present work aims at presenting the current SatCen RTDI activities on the usage of Sentinel data for Security applications, in particular of Sentinel-1 and Sentinel-2 data. The added value of these Medium-High Resolution satellites to the established image analysis practice in the Space and Security domain, based mainly on Very High Resolution (VHR) data, is the ability to provide quick views of large areas as well as to support pro-active monitoring of Areas of Interest (AOIs) in view of future tasks. Based on these assets, an initial set of prototype services was identified and developed in the frame of RTDI projects and initiatives.

Sentinel-1 SAR images offer a valid support to current interpretation analyses due to their insensitivity to weather and illumination conditions as well as to the effectiveness for the detection of objects with strong backscatter relative to the background; as an example, these data can be useful for the monitoring of critical infrastructures such as harbours and airports. A pre-processing service for Sentinel-1 GRD data has been developed and refined, where a chain of SAR image correction and calibration modules (based on SNAP operators) allows obtaining an orthorectified image to be visualized or downloaded for further analysis with external tools. Based on this service, an Amplitude Change Detection (ACD) service was developed, where pre-processed Amplitude SAR images are compared and a Change Detection map is derived. Moreover, advanced techniques based on coherence computation allow to detect small changes on the Earth surface, often undetectable with the only use of optical data; as an example, loss of coherence can be used for the monitoring of non-paved migration routes. Sentinel-1 SLC data were therefore used to create a Multi-Temporal Coherence (MTC) product, where the coherence image from a pair of SLC images is jointly visualized with the Amplitude of the Master and Slave images in an RGB complex.

Sentinel-2 data are optical data offering the possibility to monitor an area with high revisiting time in several regions of the Electromagnetic Spectrum (VIS, NIR and SWIR). These images are particularly useful to display variations at relatively large scales of Land Use / Land Cover (LULC), for instance to identify areas with manmade structures as well as areas with potential illicit crop production or illegally deforested. These data are also useful to display variations of different surface characteristics, for instance to assess flooding events threating the security of citizens. Sentinel-2 data pre-processing (e.g. atmospheric correction and super-resolution) is currently explored as an automatic service to prepare an image for further analysis. Such analysis can be performed through the extraction of thematic indexes, based on combination of bands (e.g. water indexes to identify flooded areas and mineral indexes to identify mining activities) or from suitable change detection techniques.

Finally, the possibility of generating a stack of Sentinel-1 and -2 data allows the creation of a time series through a multi-band raster product, where the user can exploit the synergy of using SAR and optical data together (e.g. to create a continuous timeline of events, where SAR data complement the analysis in cloudy images).

These applications have been developed in the framework of H2020 projects and other RTDI initiatives as the development of a platform for geospatial data management. In particular, Sentinel-1 pre-processing and ACD services have been developed in the framework of the BigDataEurope and EVER-EST projects, while Sentinel-1 MTC is being developed in the framework of the BETTER project and internally validated. Sentinel-2 studies are currently performed within the NextGEOSS and BETTER projects using an indexes service internally deployed. The final aim is to transfer the knowledge acquired in these activities to increase SatCen and its Stakeholders’ operational capabilities.

Road networks has always been of major importance for defence and security applications. They connect people and assets, ensure mobility and enable the exchange of goods. They also serve as supply routes for military operations or migration and smuggling routes. Hence, an up-to-date overview of road networks is crucial for security applications concerning awareness and understanding as well as crisis prevention, preparedness and response.
Available road data often lacks in reliability, completeness, homogeneity and currentness. This is particularly apparent for local networks in remote regions as experienced using OpenStreetMap data in North Africa and India. An alternative source for road network data is the large and steadily increasing amount of satellite data. Thus, the development of suitable methods for automated information extraction is becoming indispensable. Recent advances in the area of deep learning have successfully contributed to automating traditional object extraction tasks.
A common approach to extract features is to manually segment or vectorise the images. As a time-consuming task requiring a high level of expertise, this approach does not scale well to the drastically increasing amount of satellite data available combined with the likewise increasing demand of frequently and continually available up-to-date geodata in order to gain a timely awareness of a situation. Most established approaches to automate the process of object recognition are limited to specific scenarios (e.g. seasons, climate zones) and satellite specifications (e.g. spatial and radiometric resolution, viewing angles). Due to these limits in generalization capability, they often fail to meet requirements of time-critical security applications.
Novel network architectures proposed in the field of deep learning, such as U-Nets, received particular attention for their performance in pixel-wise semantic segmentation and their generalization capabilities. They have been applied successfully to one-class segmentation tasks, such as road detection, but as well to multi-class segmentation, including e.g. vegetation, buildings.
The present approach applies deep learning for road detection towards leveraging applications like situation awareness or disaster mapping. For this purpose, the models need to (a) generalize well to different satellite scenes with different spatial resolutions, geographic regions, seasons, (b) be efficient in terms of training and prediction time, and (c) allow for both raster and vector data output for subsequent processing.
In order to meet these requirements, an effective workflow for automated road extraction using a deep learning approach was implemented. The workflow as illustrated in Figure 1 consists of two main phases: prediction and post-processing.
Very high-resolution satellite images of different regions of the world were used for the training data generation. Several AOIs were selected and labelled manually, guarantying a very high accuracy. The AOIs contain both, rural and urban areas of 11 regions in 3 continents (varying in e.g. season, climate zone, urban arrangement).
This deep learning approach applies a well-known convolutional neural network architecture (U-Net). Several models are separately trained on images containing urban and rural like regions. Afterwards, the predictions of all applied models is combined, allowing an accurate segmentation in both, rural and urban areas.
During post-processing, another network of the same architecture but trained on thinned targets (centrelines derived from vector data) is applied. A following centreline algorithm and a custom simplification algorithm transforms the results to a connected vector representation of the road network. Figure 2 shows exemplary the outcome of the major workflow steps.
To already investigate the transferability of the developed algorithm to other sensor specifications without previous training data generation and to evaluate the potential of an abstraction to another scale, satellite imagery with a lower spatial resolution of Planet Dove and Sentinel-2 were exemplary tested.
Figure 3 compares both results. The two pictures on the left side show the results as trained for very high-resolution imagery, while the two pictures on the right side display the outcome of the algorithm as applied on non-trained imagery data of a lower spatial resolution.
This leads to the following conclusions:
(a) The more training data are implemented into the system, the more reliable and suitable are the results. In the current phase of the project, the majority of training data were situated in urban areas with well-approved road conditions. Thus, the best results were reached in such areas (see results where WorldView data were used). The recognition rate especially in urban areas is very high and networks are closed.
(b) The abstraction of the model to sensors with a lower spatial resolution without the usage of specific training data is already suitable. As seen with the example of Planet Dove data, the recognition rate is still high but with first missing network connection occurring. Even the use of Sentinel-2 data in rural areas with a strongly differing spatial resolution compared to the training data provides the user an overview and a first impression of the road/ track network of the area.
(c) The current model allows a potential application on worldwide level disregarding influencing factors such as season, climate or vegetation zone. Up to now, the applicability is widely possible even that only certain regions are trained. More training with a wider input would further robust the model.

Further generation of training data cannot only increase the accuracy of the results and widen the applicability on further geographical regions, but it may as well enlarge the suitability and extension to further sectors.
Due to the increasing amount of satellite data and the rapid change of infrastructures, it becomes extremely relevant to automate the process of object extraction. The current workflow enables road extraction from satellite imagery in a very fast and flexible manner. Further, the detection and generation of road networks in both formats, raster and vector, enables a widespread use in various thematic fields of application and sectors which can be exemplary named, among others:
(a) Network analysis: considering e.g. network density, surface conditions, network length.
(b) Change detection: Monitoring of existing networks regarding development or changes as well as analysis of activity.
(c) Mapping applications: covering various fields such as topographic mapping, rapid mapping, disaster mapping.
(d) Support to a common operational picture (COP) to achieve situational awareness.
The automation of road extraction implies the prerequisite to provide security applications as a service. The portability of our approach to different objectives (e.g. helipad monitoring, camp analysis, etc.) will help to extent the portfolio of security applications and support defence and security agencies to face the challenges of today and the future.

During the last 10 years in Sub-Saharan countries, many insurgencies have developed in borderlands, like in Mali, Burkina Faso, Nigeria, Niger, Cameroon and Chad. In these countries, some borders are "fragile": the authority of the State is challenged, State actors sometimes abandoned the area and armed groups use the border as (i) an economic resource for taxation and logistics, (ii) a political resource as an evidence of the State's failure to impose its governance at the national boundaries that are a highly symbolic part of the national territory, and (iii) as a geographical heaven complexifying the action of security States' actors who are obliged to cooperate to operate.

These armed groups like AQIM, IS in West Africa, Boko Haram, IS Great Sahara operate locally in and from the borderlands, but within a transnational ideological framework of future expansion. Locally, they try to create a political order, as any governance actors, and, outside of their settlement area, they try to propagate disorder through terrorism, in particular in rich countries.

The proposed paper is based on 15 fieldwork missions in fragile borderlands in SSA, in cooperation with local Customs administrations and security and defense forces, between 2016 and 2018. It will focus on the case of Boko Haram in Lake Chad Basin to (i) illustrate to which extent geospatial data is critical (and locally lacking) in the fight against insurgencies, (ii) propose geospatial data and tools for spatial risk analysis conducted by State border agencies and (iii) explore the use of satellite imagery (open source and commercial) to set up a realistic scheme for spatial intelligence in borderlands and (iv) and share outcomes related to the detection of suspect areas.

The paper will present the spatial modelling of the Area of Interest, how this modelling relies on and promotes a specific perspective on the border based on movement and wealth circulation, how this modelling can be used for spatial risk analysis and the detection of suspect areas. The comparison between these modelling outcomes and the satellite imagery will be presented as well as how satellite imagery helps to adjust the modelling and the spatial risk analysis.

By using geospatial data and tools to represent the border through an economic modelling, we aim to coin the economic dimension of insurgencies within the current security policies, which is critically missing today.

This research is part of a project on Fragile Borders, supported by the World Customs Organization for its 182 State Members, aiming to foster the action of civilian State border agencies and reinforce their cooperation in security policies.