Following its invention of datacube services [Baumann 1992], the rasdaman team since then has architected the rasdaman scalable datacube engine for cross-domain management and server-side processing of massive n-D arrays. The rasdaman declarative datacube analytics language allows any query, any time, on any size, ready for extraction, filtering, processing, analytics, distributed datacube fusion, and Artificial Intelligence. Through the OGC WMS, WCS, and WCPS standards rasdaman supports regular and irregular spatio-temporal grids. The open-source rasdaman engine is OGC WCS reference implementation, and the rasdaman query language has been standardized in the ISO SQL array extension. As the currently only tool rasdaman supports transparent federation on planetary scale. Individual nodes retain autonomy, with security configurable down to single pixel level.

The rasdaman architecture is a full-stack implementation in fast C++, with every component optimized for arrays. A series of highly effective techniques boosts array query execution, including: adaptive partitioning, just-in-time compilation into machine code, heterogeneous CPU/GPU hardware support, intelligent query optimizations, parallelization and distribution (without single point of failure), and federated processing. Petabyte datacubes have been established with rasdaman, and queries have been split across more than 1,000 Amazon cloud nodes.

Storage utilizes transparent tiling into sub-arrays maintained in some database (like PostgreSQL) or directly in the file system (about 2x faster), always using rasdaman’s optimized storage methods. Any tiling can be configured by the admin, as one of the many tuning parameters in an easy-to-use, OGC based ETL tool. Additionally, rasdaman can tap into any pre-existing archive by just registering files rather than copying them.

With Athena Research’s FeMME extension, rasdaman allows integrated data and meta-data search, thereby establishing a seamless integration of catalogs with datacubes. Users always remain in the comfort zone of their well known tools – rasdaman supports a large and growing number of visual and API clients, including OpenLayers, Leaflet, QGIS, ArcGIS, NASA WebWorldWind, Cesium, C++, Java, python, and R.

Altogether, rasdaman aims at user-friendly, flexible, fast, scalable, and standards-conformant datacube services, so that providers enjoy simple deployment and maintenance, users enjoy the massively simplified access, visualization, and analysis capabilities not requiring any programming skills. A series of innovation awards, most recently the NATO Defence Innovation Award 2018, acknowledges the rasdaman innovation. Many projects and services utilize rasdaman, such as CreoDIAS, CODE-DE, the EURAC Alps Datacube, etc. (see Figure for a kaleidoscope of rasdaman-enabled EO portals). Further, rasdaman is the common platform of the emerging European Datacube Federation.

[Baumann 1992] P. Baumann: Language Support for Raster Image Manipulation in Databases. Proc. Int. Workshop on Graphics Modeling, Visualization in Science & Technology, Darmstadt/Germany, April 13 - 14, 1992, Springer 1993, pp. 236 - 245

Area D - Board 370

The EO market is in a rapid expansion and, more and more, EO derived products and services will be part of value chains different from their original ones. It means that each single step of the EO services value chain could be part of different market sectors. The more it happens, the more EO market needs to improve its maturity.
Some fundamental steps towards the maturity are represented by the authenticity of the managed information and data, and the integrity of each processing step applied to them, from their acquisition to their delivery to the customer or to the Service Provider.
In fact, from the first appearance of the EO satellite data inside an archive under a cyber-attack, the satellite data can be stolen, damaged and substituted by fake copies that can invalidate all the information to be extracted from them for all the customers and the EO service providers. That, of course, can occur in each processing step of the value chain.
We, as SME active in the space market and being part of the value chain, consider that the addressing of these issues is strongly becoming a necessity for all the players. This vision is shared with ESA that recently awarded us with a contract to investigate on the application of suitable technologies to EO.
Those problems can be addressed in different ways, but a promising technology like those that fall under the name “blockchain” seem the best fitting solution to that. Moreover, from a technical point of view, the DLT (distributed ledger technologies) is simply a database relying on three fundamental concepts: Peer-to-peer networks, Public-key cryptography, and Distributed consensus.
When dealing with satellite imagery, we have to cope with very large datasets and lots of metadata. Proprietary algorithms elaborate those datasets to generate value added products, which EO companies and institutions deliver further in the value chain to the final users. Data integrity, value chain participants’ identity and reputation, data freshness, processing reliability and cyber security in general are the most important concerns to settle for stimulating downstream industry growth and overall service quality and reliability.
We identified public key cryptography, peer-to-peer networking, distributed consensus and incentive based operations one of the proper solution to guarantee: Identity (the source and destination endpoints of the data), Integrity (the data was not counterfeit), Freshness (the data was processed and the time-relation between original and resulting data was stored and certified), Overall data and transaction security.
We then propose a thin application layer (a software library on dedicated hardware) that allows any client application to participate to a peer-to-peer network with the aim of trading specific EO content by using a customized incentive based (by crypto currency or rating mechanics) distributed smart contracts executive (sort of distributed execution cloud).
The EO content is represented by any data related to Earth Observation activities, be the data the original satellite produced imagery, any value-added product (VAP), specific product metrics and metadata.

Area D - Board 357

In the last decade we witnessed a massive technological explosion in Computer Science, particularly the Cloud gained increased attention both inside and outside of the IT community. Unfortunately this new developments have yet to be adopted by the Earth Observation data processing community. Particularly most existing, open source, software solutions are still designed for standalone operation or for supporting legacy cluster workload managers. In this context we propose the EWPS server aimed in integrating the latest innovations in the cloud ecosystem with existing OGC standards.

The EWPS server presented was developed as part of the ESA funded EO4SEE Pathfinder project and it was designed to be a standalone WPS 2.0 server.. As demonstrated during the project, EWPS provides a user-friendly way of defining and describing WPS process, by introducing a YAML based mapping of the WPS 2.0 concepts. In addition to this, we have improved the process description mechanism by formulating a new Python API, based on multiple Python specific features (e.g. decorators, type annotations). As opposed to existing WPS 2.0 implementations, we focus on bringing modern job schedulers for handling the WPS job execution process. We integrate three different, customizable, job execution platforms: a local Process Executor, the Mesos based Cook Job Scheduler and HTCondor. The Local Process Executor allows local job execution, by making use of the Python Threading and Multiprocessing mechanisms, supporting both synchronous and asynchronous process execution, as specified in WPS 2.0 standard. Cook Job Scheduler comes as a resource manager over Apache Mesos, enabling the execution of Docker processes. We also integrate HTCondor, a specialized batch scheduling system for managing compute intensive jobs. HTCondor was designed to make use of non-dedicated machines to run processes, managing the available resources and scheduling the job execution. Regardless of the job scheduler, we offer a unique result management, storing the processes results using an artefact management datastore. We support disk-based storage, providing transmission by value and by reference of the output result, as specified by the WPS 2.0 standard.

EWPS was designed to be easily extended, allowing easy integration of new execution and storage backends. In this context we aim to add support for new technologies like Kubernetes and BLOB storage (Swift, S3). In addition to the aforementioned features, the EWPS aims to comply with Testbed-13/14 proposed extensions. Testbed-13/14 Earth Observation Clouds (EOC) proposes a new strategy for WPS processes, enabling them to be executed independently, in a predictive manner on different cloud providers. This is done by introducing the Application Deployment & Execution Service (ADES) component, responsible for registering new WPS processes packaged as Docker containers. EWPS also supports the Testbed-13/14 introduced Central Geospatial Catalogue, as a source for retrieving process inputs. Catalogue details are mapped in the OWS Context Document and are used to extract the proper process inputs before starting the execution. From this perspective EWPS is able to connect with any WPS 2.0 compliant Application Management Client Interface (AMC). A proof of concept of ADES and Central Geospatial Catalogue integration with EWPS was also presented during the OGC EOCP Hackathon.

Area D - Board 358

To enable the development of Earth Observation (EO) analysis by industry and government, Geoscience Australia’s Digital Earth Australia project has developed an open source Infrastructure as Code architecture that utilizes the Open Data Cube (ODC) to deliver Cloud Optimized GeoTIFFs (COGs) through Open Geospatial Consortium (OGC) API standards. This infrastructure enables EO data to be used to make decisions by industry and government partners, and reduces the time required to deliver new EO data products.

The ODC is an open source project that is used to manage and analyze EO data. The ODC enables Digital Earth Australia to perform analysis of EO data and generate derivative EO data products for use by industry and government partners. Over the last year, Digital Earth Australia has invested in building cloud infrastructure using open source technology to deliver our EO data products to the public. The development of this cloud architecture has enabled the Digital Earth Australia project to serve over 200TB of EO data products to thousands of concurrent requests.

This talk will cover the open cloud agnostic tools that enable the delivery of Digital Earth Australia’s EO data products through cloud services. This includes the file conversion and upload pipelines that deploy new EO data products and update existing products in our open data repository. It will also describe our Kubernetes based orchestration system that presents this data as OGC compliant web services using containerized ODC OGC web services. This system has enabled the quicker release of EO data products in the formats that our industry and government partners prefer.

This presentation will showcase the advantages and disadvantages of our infrastructure and highlight the benefits of using open and tested architecture patterns to deliver EO data products to users. Through discussing our open source infrastructure, we hope to share our experiences with other organizations enabling them to leverage our architecture and code to deliver EO data to their partners. Ensuring that government and industry can easily leverage EO data to assist in their decision making.

Area D - Board 359

Data cubes are among the most important facilitators of big Earth data analyses. They hide the complexity of data management from users behind a standardised query interface or an API, and allow queries on the temporal dimension equally to the spatial dimensions. A data cube supersedes the traditional notion of a scene, a 2-dimensional image of the Earth's surface, and instead treats individual pixels as observations, which are located within an N-dimensional data structure without artificial extents, such as a scene border or the temporal snapshot model implied by scenes.
However, we found several requirements or limitations in state-of-the-art data cubes. In general, combining data with different sampling frequencies is difficult, e.g., MODIS pixels with 250 m and Landsat pixels with 30 m ground sampling. Furthermore, Object-Based Image Analysis (OBIA) improves the expressivity of analyses significantly by applying a GIS-oriented view, allowing users to locate image objects, to count them, to structure them, and to put them in spatial, temporal, or semantic relationships. Moreover, many implementations require to fix the schema, e.g., the number of bands, or properties, during the initialisation of the data cube. Although common software packages, e.g., Rasdaman or the Open Data Cube, are being constantly improved, we freely revisit and re-think concepts and properties, which are inherent to Earth observation (EO) data. In our approach, we revisit the geo-atom concept as smallest unit of geographic information and transform it into a smallest observed geographic aspect of an entity. This GIScience-based consideration is necessary, as the original geo-atom describes the smallest geographical information, while in EO, an entity can never be fully observed, but only aspects of it. Atoms associate a geographic location to a property as a tuple (x, g, Z, z(x)) where x is the location in space-time, in this case coordinates, g the grain ("resolution"), Z a property, and z(x) the value of the property Z at that location x. Our data cube approach aims to provide an integrated information space holding any type of atoms (tuples), from different sensors, at different scales, different properties, etc.
While a data cube inherently maintains grids of pixels, the atoms need to be put in a relationship to each other prior to the analysis. Although this creates additional overhead, it allows the flexibility to define the relationship on a case-by-case scenario for each application. Therefore, it is possible to group them into a coverage ("field") of features ("objects"). A coverage is treated as a spatio-temporal varying attribute of a feature, which contrasts spatio-temporal uniform attributes, e.g., a name. For example, the feature "Atlantic ocean" might have the spatio-temporal non-uniform salinity or sea surface temperature as coverage attribute. By grouping atoms based on their property values, it is possible to create child-features, which are in a part-of-relationship to the parent feature (the study area is always the first feature). Some of the benefits, which come with a feature-based view on data cubes are traditionally known from GIS:
- referring to parts of an object using the feature's semantic type (e.g. "lake")
- counting features
- locating features within the image
- group features into complex types
- estimating beginning and ending of features ("lifespan"), which contrasts the "change-of-colour" in the coverage-based view, where begin and end is not a concept
- combining different datasets, including sensors with different resolution, in-situ data coming from objects moving through space-time.
At a technical level, objects are a set of atoms, while a data cube is a set of objects. Initially, every atom build its own object. The objects can be selected and grouped into to larger objects. Atomic and non-atomic objects are treated the same way, i.e., also non-atomic objects may be grouped into new, larger objects. The grouping procedure may use using different concepts of neighbourhood. This might be Cartesian distance, but might also be more complex spatio-temporal relationships.
We present a first application, based on the Open Data Cube, where we group distinct agricultural fields in Egypt based on their high NDVI values in contrast to the surrounding environment and run both typical coverage-based queries (e.g., integrate average values or calculate the histogram) as well as typical feature-based (GIS-based) queries (e.g., calculating the size or density). While the feature detection itself is not the main focus of this work, our aims include the development of a query language and API combining the coverage- and feature-based view, a tight integration into the open data cube, and increase the performance by using intelligent indexing using space-filling curves.

Area D - Board 360

In order to support users in extracting value from data from Earth Observation satellites and performing research involving trend-analysis over long periods of time, the European Space Agency has specified use cases, requirements and an architecture for a Heritage Mission Valorisation Environment (HM-VE). Heritage data are extremely valuable for long time-series studies such as those concerning climate change, due to their uniqueness, temporal coverage and the impossibility of collecting further observations back in time.

The paper highlights how Heritage Mission data is particularly relevant when its Value, Variety and Volume (three of the Vs of Big Data) are taken into consideration. Such characteristics are enablers for climate studies and long time-series, but they also pose unique challenges that any infrastructure aiming to deal with them needs to address.

The paper then presents the main requirements for the environment, as well as the current status of its implementation, emphasizing how ongoing investments - by ESA and by several other players - in datacube technology play a role, particularly for the use cases related to data access, visualization, analytics and even hosted processing.

Specifically, the paper goes into detail about a prototype datacube implementation for ESA Heritage Missions - namely ENVISAT and ERS -, which manages Level-0 and Level-1 data and allows users to interact with it, with no need to download anything ("bringing users to the data" paradigm). Users can browse collections and define Areas and Times of Interest using a GUI or create/edit interactive notebooks (e.g. Jupyter) to browse, access, process and visualize the data through live coding. The same API can be exploited from a terminal by advanced users.

The interactive notebook functionality and API make use of OGC standards such as WCS and OpenSearch, which bring additional value to the prototype, not only due to the generic benefits and power of these standards, but also when looked at from the federation point-of-view. Indeed, the implementation is prepared to integrate a federation of datacubes using the same technologies, made available by several organizations worldwide.

The paper concludes by explaining how datacubes and this prototype implementation are being harnessed for the implementation of the HM-VE.

Area D - Board 361

The Africa Regional Data Cube (ARDC) is a cloud-hosted system based on Open Data Cube (ODC) that harnesses the latest Earth observation and satellite technology to help Kenya, Senegal, Sierra Leone, Ghana, and Tanzania address food security as well as issues relating to agriculture, deforestation, and water access. The system was set-up, configured, and deployed by the Committee on Earth Observation Satellites (CEOS) System Engineering Office (SEO) in partnership with the Group on Earth Observations (GEO), Amazon Web Services (AWS), Strathmore University in Kenya, Office of the Deputy President - Kenya, and the Global Partnership for Sustainable Development Data (GPSDD). Earth observation data and satellite imagery assist researchers studying food security to help end hunger in Africa. The ARDC can also play a critical role in supporting refugees, internally displaced peoples, and people living in fragile or conflict-affected states.
The formal launch of the ARDC, and an in-depth hands-on training workshop for over 60 government representatives of the five participating countries (Sierra Leone, Ghana, Senegal, Kenya and Tanzania) took place in Nairobi, Kenya on May 9-11, 2018. The AWS-hosted ARDC User Interface (UI) and Jupyter Notebook Server were made available along with 15+ terabytes of satellite data for the five African countries. The attendees participated in several interactive tutorials that illustrated how easily the ODC facilitates time series analyses (e.g. for agriculture, water, forests, disasters), the use of diverse datasets with interoperable methods, and connections to common GIS tools (free or commercial). They learned how to use the User Interface (UI) and Jupyter notebooks, perform time series analyses, and work with image compositing, water detection, and land change applications.
During the workshop, the government representatives of the five participating countries placed great emphasis on the fact that Africa has recently emerged as one of the fastest growing markets for mobile phones in the world. They also brought to everyone’s attention that farming is the primary source of food and income for Africans and provides up to 60 percent of all jobs on the continent. Food production in sub-Saharan Africa needs to increase by 60 percent over the next 15 years to feed a growing population. In a nut-shell, they identified the need and demand for the development of an ARDC Mobile App to provide farmers with access to useful information from the ARDC in an easy to understand format. In keeping with the goal of using satellite data to address issues such as food security, agriculture, and water access, a cross-platform mobile application designed to put Earth observation data directly into the hands of the people was required.
In today's world, Internet-enabled mobile phone use has revolutionized many aspects of daily life, and the increasing availability of such devices, even within developing regions of the world, has the potential to boost efficiency in unforeseen areas, such as agriculture. For small farmers, especially in developing countries, these devices may provide extraordinary benefits including income growth, decreased waste of resources, and larger yields. In recent years, several mobile-based applications have been deployed with the aim of assisting small farmers in a variety of localities, most of which focus on equipping the farmers with market, crop, and environmental information, best agricultural practices, or records management. Many of these apps are built on a real-time architecture with strategies for handling bandwidth and connectivity issues.
Farmers living in remote areas face difficulties obtaining necessary supplies and selling their commodities. Oftentimes, they are subject to losses due to the greed and corruption of middlemen, spoilage during travel, and limited market knowledge. Since inputs required for agricultural production, such as fertilizer, seeds, loans, etc. and markets for the sale of final agricultural outputs are typically far distances from these farmers, they often rely on mediators who also seek to make a profit.
There are some information database apps developed for small farmers, such as iCow and Tulaa. iCow, like Tulaa, was originally designed to disseminate agricultural information via SMS. In August 2017, the iCow service had 50,000-60,000 users in eastern Africa who typically experienced increased productivity in 3 months. In July 2018, iCow released a mobile application which allows access to their platform as well as audio recordings. Knowledge stored on the iCow platform enables farmers to perform a geographic search to select the best corn seed, learn about their regional soil conditions (pH, total organic carbon, and available nutrients), and to connect with soil testing services in their area. They may also access a general library of reference materials or conduct a symptomatic search of crop diseases, allowing them to diagnose and remedy issues they may not have known about prior to using the app. These educational subscription packages are commended as the largest contributor to the enhanced productivity of the farmers.
It is envisioned that with drones, farmers may obtain various metrics that indicate overall crop health. Sentera offers many solutions that couple with a mobile app and desktop software. From the app, farmers may control a drone and view output from the drone. Imagery that may be retrieved from the drones include standard RGB, Normalized Difference Vegetation Index (NDVI), Normalized Difference Red Edge (NDRE), and 3-D projections. From these products farmers may build zones for regulation, improve water management through elevation mapping, create an unlimited number of mosaics, estimate crop populations, and identify areas of weed infestations. All this information can be exported and shared with others or even agricultural equipment for further management, like fertilizer and pesticide application. Moreover, data can be stored for comparative analysis with future crops.
By taking all the valuable feedback from the community into consideration, the CEOS-SEO software development team developed the ARDC mobile app. The software is designed to bring USGS data from the ARDC to the fingertips of the user in an easily accessible format. The scope of the application will expand in stages, beginning with a large portion of Kenya, expanding to the entire region serviced by the ARDC, and eventually connecting with any Open Data Cube instance. Currently, users select points within the Kenyan region to analyze, and the application displays NDVI and Water Observation from Space (WOfS) data at that location. Data is displayed in time series graphs that can be altered by the user. All the data is overlaid on a map powered by the Google Maps Application Programming Interface (API). The ARDC Mobile App is scheduled to be released by the end of 2018.
Many design decisions were made after careful requirements analysis. The ARDC app interacts with a server-side backend, web services, and APIs. These APIs are provided by the ARDC and/or reliable third-parties. The application itself was developed in React Native, which was chosen because it is an industry standard cross-platform environment. Its use of JavaScript simplifies development and makes it easy to incorporate into future projects. React Native was specifically chosen out of a pool of cross-platform frameworks for mobile development. The availability of a large set of libraries compatible with React Native and a strong community support for it make it a strong candidate for adoption for such mobile development. Several providers of interactive mapping software were evaluated, and Google Maps emerged as the leader over e.g. Mapbox due to its performance in rural areas, which is the primary scenario for our app. The Victory Native library was chosen for graphing because it is one of the most popular and well supported graphing tools in the React ecosystem.
The ARDC Mobile App has the potential to transform agriculture in Africa. This app will be open source, free, and will work on very basic smartphones with limited network connection. Through context-based data caching, it will be designed to work with no or intermittent network connection once the app has been downloaded. In developing countries, where the vast majority of the national food supply is produced by small farmers, taking advantage of technologies that empower these farmers is critical in order to better address food shortage crises.

Area D - Board 362

Over the last few years, an extensive set of technologies have been systematically included in precision agriculture (PA) and also in precision viticulture (PV) practices, as tools that allow efficient monitoring of nearly any parameter to achieve sustainable crop management practices and to increase both crop yield and quality. However, many technologies and standards are not yet included on those practices. Therefore, potential benefits that may result from putting together agronomic knowledge with electronics and computer technologies are still not fully accomplished. Both emergent and established paradigms, such as the Internet of Everything (IoE), Internet Of Things (IoT), cloud and fog computing, together with increasingly cheaper computing technologies -- with very low power requirements and a diversity of wireless technologies, available to exchange data with increased efficiency -- and intelligent systems, have evolved to a level where it is virtually possible to expeditiously create and deploy any required monitoring solution. Pushed by all of these technological trends and recent developments, data integration has emerged as the layer between crops and knowledge needed to efficiently manage it.
In this work, the mySense environment (MSE) is presented, aimed to systematize data acquisition procedures to address common PA/PV issues. MSE is an innovative data integration platform specifically created and continuously developed to provide solutions to specific problems of PA/PV. This platform relies on a cloud-based infrastructure that supports the development of particular solutions based on static, mobile, proximity or remote static sensors to provide essential data to support a wide variety of agroforestry practices. Simultaneously, machine learning algorithms are used on data from these sources, aiming the implementation of decision support tools.
All devices associated with MSE produce data that can be numeric and/or images with support to geographic referencing, which allows the association of data to a location context. These data are indexed to a user or to group domains and considers the needs of each one assuring, at the same time, data privacy. In addition, MSE users are free to make the data produced by their own sensors available to the community, thus contributing to a richer and more solid knowledge database. Through MSE, this work aims to address the development of agricultural or forestry crops, including certification, plant health assessment and prediction of disease/pest occurrence.

Figure 1. Overview of the mySense Environment
Keywords: Precision agriculture; Precision viticulture; Smart Farming; Data integration; Internet of Things


Acknowledments
The authors would like to acknowledge ERDF and North 2020 - North Regional Operational Program, as part of project “INNOVINE&WINE - Vineyard and Wine Innovation Platform" (NORTE-01-0145-FEDER-000038) for partial funding as well as the ERDF - European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within project POCI-01-0145-FEDER-495 006961, and by National Funds through the FCT - Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) as part of project UID/EEA/50014/2013.

Area D - Board 363

Environmental data volume is growing every day. Temporal and spatial resolution is getting better and data collecting sensors and models are getting more advanced. It is clear that the amount of data grows with technology growth and it’s not stopping any time soon. All this data is shared in different sources and is often very poorly documented and distributed by different protocols. Because all of this, data preparation and processing usually take the most of the time before any analysis even starts.

Planet OS Datahub by Intertrust (data.planetos.com) provides seamless API access to 380+ curated weather and climate parameters from the world’s leading open data providers via convenient formats like JSON, CSV or netCDF, so professionals can stop struggling with data integration and focus on creating value for their communities and stakeholders. We are acquiring data for historical analysis as well as operational forecasts to cover current weather events.

When we find an interesting event through exploring the data, we share it with the world. Whether it’s looking into air quality during California wildfires, forecasting snow conditions in Winter Olympics, or how climate has changed down under in Melbourne in the last 100 years – we accompany these stories with free code samples and instructions, to provide our community with tools to tell their stories better. We share our stories in Medium - https://medium.com/planet-os.

At the moment, Datahub has 86 open datasets, containing about 12 TB of data and including around 380 unique standard names (by CF convention). All the datasets are free to access, with up to 100 API calls per day. Each dataset has detail page with metadata and data preview for every variable. Datahub provides search API and Web UI so users can easily find any published dataset via keywords and filters like location, variable names, resolution, products type and so on.

We are open to collaboration on publishing new datasets and support custom commercial solutions. Our goal is to make data access as effortless as possible, so, we keep releasing Jupyter Notebook examples (https://github.com/planet-os/notebooks/tree/master/api-examples) to show how to use the data and perform data analysis comfortably.

Area D - Board 364

The launch of recent satellite missions, the Sentinel fleet of the European Copernicus programme in particular, has led to a tremendous increase in available earth observation data provided at no cost. The increase in data availability opens new opportunities for analysing data not only in the spatial but also temporal domain by observing time series and thus the possibility to visualise processes on the earth’s surface. Although this is not entirely new to optical satellite data, Synthetic Aperture Radar (SAR) data was only infrequently acquired in the past. With the new ESA SAR satellites Sentinel-1A and Sentinel-1B there is now the possibility to observe the earth’s surface with a repeat rate of up to six days and a spatial resolution of 10 m independent of atmospheric effects and sun illumination.
Together with the increase in data availability comes the challenge of organizing data and preparing it for scientific analysis. While traditional software aimed at analysing single images, the need arises for fully automated organization of large image archives with thousands of images together with a highly capable processing framework to make full use of available hardware resources.
The pyroSAR environment aims at providing a complete solution for the organization and processing of SAR satellite data for applications scalable from personal computers to large server infrastructures using only open source tools and libraries. Its purpose is to provide complex functionality for reading various data formats from past and present SAR satellite missions, handling metadata about acquisition characteristics in a database, and providing homogenised user-friendly access to processing utilities in ESA’s Sentinel Application Platform (SNAP) as well as GAMMA Remote Sensing software. The individual processing steps are recorded in homogenized XML workflows so that a dataset is always annotated with information about its way from raw source to analysis ready format.
The data reader uses the Geo Data Abstraction Library (GDAL) where possible and own implementations otherwise. The metadata attributes are homogenised to enable database access of specific acquisition characteristics across different sensor platforms. The metadata is ingested into a database from which the original imagery can be selected for processing whilst keeping track of what had been processed before.
The SAR processor provides functionality to distribute the tasks across different computing cores as well as different server nodes. By following a stringent naming scheme for processed images as well as annotated metadata XMLs, processing can be organized to be performed by several operators in a server network. This way, redundant usage of disk space and processing resources can be reduced.
Once the images are processed, further functionalities are available for mosaicking and resampling images to common pixel boundaries suited for time series analysis or directly exporting the imagery to an Open Data Cube.
The overall aim is thus to reduce the entire process of image processing to very few commands in order to achieve maximum usability. By simply providing a test site geometry, the type of data required and some additional parameters, like spatial resolution and time frame, all available raw SAR-data will be retrieved, pre-processed and exported to a selected format. Thus, the time needed for handling data and software, which could otherwise be spent directly on the analysis of data and development of algorithms to derive information from the data, is greatly reduced.

Area D - Board 365

Change detection is one of the important tasks in Earth observation and monitoring. Analysing the changes in synthetic aperture radar (SAR) and multi-spectral optical images through different time points guide us in discovery of significant environmental events, and in managing forest and agricultural lands. In this paper we present two standalone software, WISHARTChange and MADChange based on two well-known change detection methods, the omnibus test and the iteratively re-weighted multivariate alteration detection (IR-MAD), respectively. The omnibus test method deals with a time series of SAR data and computes a sequence of test statistics for covariance matrices. IR-MAD deals with multi-spectral optical images and computes the changes between two time points.

Given the availability of Earth Observation (EO) data from different sources, the standalone software is developed to handle different formats of images such as Georeferenced Tagged Image File Format (GeoTIFF), and ENVI binary image coupled with a header file. Moreover, to overcome the big EO data challenges, the software computes the changes in two different processing modalities: 1) reads the whole images into local memory, and 2) treats the data line by line. In addition, due to independent computations of test statistics in WISHARTChange software, we adopt a parallel computing scheme.

The software is published on "https://github.com/BehnazP/DataBio" in two versions; GUI app and command-line executable. The implementation is done in MATLAB, however the software runs "without" MATLAB being installed on the local computer. The user only needs to install the MATLAB Runtime which is provided with the software and it contains all the functions needed to run.

In both WISHARTChange and MADChange users are required to provide SAR or multi-spectral optical images and can choose processing modality. More specifically, in WISHARTChange, equivalent number of looks, polarization type and names, and time points name must be specified. In addition, there is an option to select and compute the changes in a region of interest (ROI) by providing a binary mask or by choosing the ROI interactively from a provided map. In MADChange, the name of multi spectral bands is required and a threshold as a criterion to stop the iteration can be specified. Furthermore, there are two pre-processing scheme options: 1) masking the strongest changes and 2) excluding low values related to dark regions which can be used.

Each standalone software outputs maps of identified changes in multiple formats. The WISHARTChange software outputs a table containing average no-change probabilities and a figure containing three maps: first change, last change and frequency of the change. The MADChange software outputs a figure showing the probability of no-change and the canonical correlation convergence plot. Moreover, the IR-MAD variates are saved for further analysis as an image with same number of bands and same format as the provided images.

The maps of detected changes provide a better insight into analysing and monitoring of spatio-temporal dynamics for the area of the study which assists environmental managers and policy makers in decision making.

(Footnote: This work is funded by, DataBio, the European Unions Horizon 2020 research and innovation programme under grant agreement No 732064.)

Area D - Board 366

United spatial data platform for public administration (paikkatietoalusta in Finnish, short PTA) is one of the Finnish government’s priority projects and is funded thru the Ministry of agriculture and forestry. The aim of this project is to harmonize and bring together various data sources concerning spatial data collected by different government and public sector entities, to improve and enhance the transparency of data-based decision-making, create cost savings in public administration and make and make the datasets more easily available for both general public and for commercial use. The project consist of eight sub-projects each concentrating on different theme.
Satellite imagery over Finland has already been available thru various sources, but so far they have been utilized mainly by small group of professionals, as using them effectively has required some specific knowhow and equipment.
The sub-project "Distribution and processing of satellite imagery" aims to overcome these obstacles by making satellite images available in preprocessed format, transferred to national grid for ease of combining with other data sources such as national building and address data bases. Having the images available as readymade mosaics covering whole Finland and surrounding areas with radiometric and geometric correction already performed, it is hoped more people and businesses will pick up and use these data. Additional goal is to create the basis for automation of environmental monitoring services carried out by different governmental administrations.
The project concentrates on making available data from Sentinel satellites in European Commission’s Copernicus programme. Project will make available for users radar mosaic compiled from Sentinel 1 instrument three times a month on vv and vh polarizations, a cloud free sentinel 2 optical reflectance mosaic three times a year for different seasons, six thematic index mosaics such as ndvi two times a month and Sentinel 3 synergy product reflectance mosaic on daily basis.
Also provided will be three historic, cloud free mosaics from different Landsat instruments and possibility of ordering single Sentinel 2 image from location of users choosing. The last service is aimed specifically for generic public and will include original raster bands, a readymade RGB-image and two false color images, with an info package how different channel combinations can be used for highlighting different aspects of the image data.
The project will use facilities and data of the National Satellite Data Centre operated by Finnish Meteorological Institute's Arctic Space Centre in Sodankylä that hosts Copernicus Sentinel Collaborative Ground Station. The Centre has direct receiving capability for Sentinel 1 data and hosts mirror storages for Sentinel 2 and 3 data, and provides different ways to access them. The mosaics will be made available in cloud optimized geotiff format through machine readable catalog service and ordering of the single image will have its own map interface with search function connected to national address database.
The project was coordinated by National Land Survey of Finland (NLS) and the satellite image sub-project is a co-operation by Finnish Meteorological institute (FMI) and Finnish Environment institute (SYKE).

Area D - Board 367

With the constant increase of available EO data in space and time, more and more users developed the need to not only perform data processing and analysis on single EO data products and individual scenes but instead on mass data covering larger regions or longer time periods. Such activities involve heavy bulk processing of data and therefore users typically want to exploit the bundled resources of multiple connected computers at their disposal.
There is also available a variety of software toolboxes and libraries for EO data products processing, exposing many processing modules in various ways and implementations. From the community point of view, this is a good thing but it comes with a burden on the accommodation to the specificities of each such toolbox or library. Furthermore, most of these tools are designed to run on individual workstations, not allowing thus (at least not without a significant effort) the exploitation of existing computing clusters.
In order to distribute portions of work to individual machines and gathering the results, a software is needed that orchestrates clusters of computers and manages their available resources. Over the past years, numerous software projects have developed to provide solutions for this problem that exists in several application domains. There are a few such solutions in the field of EO data processing, which do quite a nice job. Nevertheless, extending such solutions and adapting them to user needs, by the user themselves, is not an easy task.
Here comes TAO. TAO (stands for Tool Augmentation by user enhancements and Orchestration) is an open source, lightweight, generic, extensible and distributed orchestration framework. It allows to reuse (i.e. integrate) commonly used toolboxes (such as, but not limited to, SNAP, Orfeo Toolbox, GDAL, PolSARPro, etc.). This framework allows for processing composition and distribution in such a way that end users could define by themselves processing workflows and easily integrate additional processing modules, without any programming knowledge requirements.
In terms of use, the TAO platform provides a mean for orchestration of heterogeneous processing components and libraries in order to process scientific data. The key features of the TAO framework consist in:
• Visual (i.e. user-done) integration of EO processing toolboxes
• Easy visual definition of processing workflows by simple drag-and-drop operations
• Integration of user-defined algorithms, written in Python or R, in the processing workflows
• Visual definition of execution topologies (collection of machines)
• DRMAA-compliant orchestrated execution of workflows
• Self-contained containerized execution on remote nodes
• User and public virtual workspaces
• DIAS compatible EO data sources abstraction for querying and retrieval of EO data
• OGC standard interfaces for workflow execution
This presentation aims at introducing the TAO framework with its features, extension capabilities and usage scenarios, and, mostly important, the ease of using it by end-users.

Area D - Board 368

In the ages of "Small Data" each data item got massaged manually by some expert in the know. With today's "Big Data" this is no longer possible (and acceptable), rather automatic acquisition, preparation, and analysis is indispensable. This imposes high requirements on the quality of data preparation, its automatic monitoring, and - based on this assured quality - automatic processing where human end users see only knowledge extracted, but remain far away from the pixels. In some sense, it is the "veracity" aspect of Big Data which still poses particular challenges.

Standardization can help here in that it defines interface "contracts" which both ends of the channel - i.e, server and client - can be made compliant with, there
by guaranteeing properties of the data (and its processing) which are adequate for the particular purpose on hand. In the realm of Big Earth Data, the concept of spatio-temporal coverages and streamlined services plays a central role. In particular, the trending concept of multi-dimensional datacubes and APIs on them is captured well. Coverages allow a concise description of all aspects of regular and irregular grids, point clouds, and general meshes. The Web Coverage Service (WCS) suite of standards addresses service APIs ranging from simple subsetting as per WCS Core up to complex analytics with the OGC Web Coverage Processing Service (WCPS) datacube language enabling "any query, any time, on any size".

Coverage standardization is progressing in several bodies simultaneously. Under good conditions, this ecosystem of datacube standards is synchronized, such as the coverage standards of OGC and ISO, as well as the service APIs of OGC and INSPIRE. Sometimes, however, divergence can be observed which is detrimental for understanding, implementing, and using the standards; examples include a deviating W3C Working Group definition of coverages and slight changes of INSPIRE over the OGC/ISO coverage definition.

In our talk we present the status of Big Earth Data standardization, with specific emphasis on datacubes. We point out commonalities and differences, mention current activities for harmonization, and point out opportunities for future enhancements to the existing standards suite. Examples presented draw upon the OGC datacube reference implementation, rasdaman.

Area D - Board 369

The availability of free and open data, such as that provided by the Copernicus Sentinel fleet, together with the availability of affordable computing resources, create an opportunity for the wide adoption and use of Earth Observation (EO) data in all fields of our society. ESA’s “EO Exploitation Platforms” initiative aims at facilitating adoption with the paradigm shift from “bring the data to the user” (i.e. user downloads data locally) to “bring the user to the data” (i.e. move user exploitation to hosted environments with collocated computing and storage). This leads to a platform-based ecosystem that provides infrastructure, data, compute and software as a service. The resulting Exploitation Platform is where scientific and value adding activities are conducted, to generate targeted outputs for end-users.
The goal of the “Common Architecture” is to define and agree the technical interfaces for the future exploitation of Earth Observation data in a distributed environment. The Common Architecture will thus provide the interfaces to facilitate the federation of different EO resources into a “Network of EO Resources”. The “Common Architecture” will be defined using open interfaces that link the different resources (building blocks) so that a user can efficiently access and consume the disparate services of the “Network of EO Resources”.
Telespazio VEGA UK - a Leonardo and Thales company will lead the definition of the Common Architecture through an open process of public discussion and consensus building with the EO community. It will be promoted as a Reference Architecture that will be designed to meet a broad set of use cases that cover Federated Identity Management, Processing & Chaining, and Data Access and Management. Leveraging free and open source software, a reusable Reference Implementation will be developed and deployed operationally, to act as a validation of the architecture and to provide an existing solution to third-parties.

Area D - Board 371

Global satellite observations have revolutionized the study of many Earth phenomena, especially in the area of natural hazards, oceanography, environmental change, weather, and climate. The number of satellite platforms and the type and volume of observations they collect will only increase in the future. Many business and scientific communities are now either starting or increasing their use of satellite data. In order to do this faster and more efficiently, they are expecting satellite data to be stored, organized, and described in a standardized manner.

Most of the data generated by Earth Observation Satellites (EOS) of Japan Aerospace Exploration Agency (JAXA) are created in HDF5 (Hierarchical Data Format version 5) format. Each mission has, however, defined its own data structure and attributes, which result in the lack of usability of the data. For example, because there are differences in the names of attributes which store time coordinates or geolocation etc., between JAXA’s satellites, users have to make a program for pre-processing to analyze these data in a consistent manner. Moreover, users can use some applications for NetCDF more easily by making some attributes of the data compliant to the standards such as CF (Climate and Forecast) convention. These are the problems which have lasted long in JAXA, and we decided to start the improvement for future satellite missions.

Now, JAXA is preparing a set of guidelines for organizing and storing data collected by JAXA satellites. The overall goal is to improve interoperability of JAXA’s satellite data for a broad range of visualization and analysis software. Specifically, the guidelines address three topics as follows:
• Encoding data and metadata in a standardized manner
• Creating netCDF-compatible files with the HDF5 library
• Reducing the size and improving I/O performance of produced files

In our presentation, we will show the overview of the guidelines, the result of the improvements of the data, and the plan to adapt the guidelines to the data of JAXA’s past and future satellite missions.

Area D - Board 372