Agricultural monitoring helps manage food supplies, children’s welfare and many other factors. Satellite data can reduce people’s workload, but field campaigns in developing countries to acquire on-the-ground reference data are needed to make this possible. This study aims to create a classifier to automatically extract crop types from drone imagery. Crop type data collected by drones are then used as training data for an automatic satellite-based crop mapping system called Sen2-Agri. The question we want to answer is how far the number of in-situ data can be reduced to still maintain the required accuracy of crop type mapping.
Our methodology is based on recent advances in image classification, which were achieved by using convolutional neural networks (CNNs). CNNs are built upon stacked hierarchical layers. The shallowest layers learn to extract the basic features from images such as edges and shapes. Deeper layers combine the features from previous layers and are able to learn more abstract objects. The final layer predicts one specific class for the object shown in the input image. Many powerful CNNs are trained to classify real-world objects of thousands of categories (e.g. dogs and cats). Instead of dogs and cats, we want to identify the crop type of agricultural plots in drone images. Since the basic features are still the same, we reuse the shallowest layers of a pre-trained CNN and only retrain the last layers to correctly distinguish between different crop types. This is called transfer learning and reduces the required amount of training data significantly.
The methodology was tested on datasets consisting of (1) summer and winter drone campaigns spread around Kasungu district in Malawi and (2) in-situ crop type data consisting of geo-referenced perimeters of agricultural plots, identifying crop types and development stage, situated within each drone capture area. Visual analysis of the drone data reveals several challenges that need to be addressed from the point of view of classification algorithm in this particular area. The crops are in different development stages (including post-harvested fields), relatively sparsely populated, which makes the soil dominating the spectral characteristic of most pixels. In significant part of agricultural plots there are tree patches or weeds in between cultivated plants.
The experiments show that the CNN-based model could classify crop types within the referenced agricultural plots with accuracy of about 80%. This is significantly better than simple convolutional networks with 7 layers trained from scratch (35%-42% overall accuracy) or Random Forest pixel-based approach (35% accuracy). We show how the results achieved depends on the amount of in-situ data and how the classification methodology can be further improved.

Field high-throughput phenotyping measurements constitutes an essential instrument for plant breeding. It includes a wide range of estimates of canopy structure and optical properties of the leaves that can be monitored during the whole growth cycle. Measurements are mainly based on continuous very high-resolution (i.e. from a fraction of mm to 1 cm) optical imagery -RGB, VIS+NIR, and LiDAR- that provides access to canopy architectural parameters. The several traits that can be accessed by these close-range remote sensing systems are presented, putting emphasis on the methods applied to transform the images into state variables. They include machine-learning approaches used for colour segmentation and organ identification as well as methods based on the exploitation of radiative transfer using models with a 3D dynamic plant architecture description. These methods are illustrated by several examples highlighting the contribution of the very high spatial resolution.
A framework is then proposed to integrate the ensemble of phenotyping measurements along with the corresponding environmental (climate, soil, crop management) description into crop functioning models. High-throughput canopy observations open an avenue to achieve a robust calibration of the model parameters, with possible access to the genetic dependency of some of them. These calibrated models can then be used to assimilate satellite remote sensing data and get diagnostic and prognostic evaluation of crop yield, and three main environmental outputs: the total biomass produced as well as the soil water and nitrogen balance. Conclusions on the research priorities to support are finally drawn.

The vastly increased availability of high resolution Copernicus Sentinel-1 (S1) and -2 (S2) data streams requires novel ways to generate high quality reference data for cross-referencing, validation, and ultimately, development of robust inference methods (e.g. machine learning) for use in near-instantaneous information retrieval for wide-area applications. This is particularly relevant in the agricultural domain, for which S1- and S2-revisits closely match the dynamics in the soil-canopy system underlying the electromagnetic signal formation. Reference data collection is a tedious and expensive task and is often limited to small scale, and specific purpose, scientific validation experiments. Only very limited amounts of standardized reference data end up in the open data domain, notwithstanding large efforts like those of the Group of Earth Observations (GEO) Joint Experiment of Crop Assessment and Monitoring (JECAM).

The use of opportunistic street-level imagery (SLI) is well-known for geo-visualization approaches (e.g. Google Street View, Mapillary). Deriving quantitative information from such imagery, e.g. for characterization of crops, is less developed, but recognised to have a significant potential [1]. The idea to “guide” SLI acquisition by integrating existing geospatial data (e.g. for routing) and EO data (e.g. to visit pre-marked anomalies) is relatively new [2]. Such guided SLI surveying can result in an increased data collection efficiency compared to conventional surveys by a factor of 7 to 8. Currently, the approach still relies on a human visual interpretation step to tag and classify the collected SLI with the observed crop type and crop phenological stage. The use of computer vision and machine learning algorithms to derive crop parameters has considerable potential to further increase the efficiency of the use of SLI in generating relevant reference data.
We demonstrate this concept with the use of SLI collected with side-looking action cameras for a fixed route survey in Flevoland (NL) on 8 dates in the growing season (March-October) in 2018 [3]. This survey was designed to ensure a statistically meaningful sample density for the main crop types and their phenological dynamics in the area and includes over 200 revisited reference parcel locations for which detailed observations were made. The SLI was collected at 1 frame-per-second (FPS) along the entire route and has resulted in a data set of more than 500,000 full HD resolution images.
We deploy machine learning using TensorFlow and a number of well-known image recognitions models, primarily based on transfer learning with convolutional neural network modules (e.g. MobileNet, Inception-v3). We demonstrate the use of incremental learning techniques with the use of models that are initially trained only with discrete crop types and then refined with additional phenology stages derived from the time series. The data redundancy in the 1 FPS SLI data set is combined with known parcel geometries to design logical filters post-machine learning to eliminate outliers and re-initiate training phases with “cleaned” samples. We show the use of a smart GUI to rapidly quality control geotagged results and inspect special cases.
The use of the machine learning results as validation data for Sentinel time series will be illustrated and compared to traditional reference data. We also discuss future perspectives, including instantaneous crop recognition using our trained models and further integration into standardized data collection protocols, e.g. those serving CAP “checks by monitoring” requirements.

[1] d'Andrimont, R.; Iordanov, M.; Lemoine, G.; Yoong, J.; Nikel, K.; van der Velde, M. Crowdsourced Street-Level Imagery as a Potential Source of In-Situ Data for Crop Monitoring. Land 2018, 7, 127.

[2] d’Andrimont, R.; Lemoine, G.; van der Velde, M. Targeted Grassland Monitoring at Parcel Level Using Sentinels, Street-Level Images and Field Observations. Remote Sens. 2018, 10, 1300.

[3] de Vries, M., van der Voet, P., van der Velde, M., Lemoine, G., d’Andrimont, R., Efficient crop phenology surveying for EO validation using guided street-level imagery, Abstract submitted to the Living Planet Symposium, 2019

In-situ data are needed to improve the land cover and land use statistics that could be derived from observations by Copernicus Sentinels. LUCAS, the Land Cover and Land Use Area frame Survey, which started in 2006, provides a surveyed sampling at EU level every three years. In 2018 more than 330.000 points have been assessed, out of these some 230.000 points have been surveyed in the field. For the 2018 survey, a new component, the ‘Copernicus protocol’, was applied to a subset of points, aiming to make the data more relevant for Earth Observation data analysis.

This study first describes the LUCAS Copernicus points sampling and surveying methodology. For these points, in addition to the standard variables registered for all LUCAS point, such as land cover, land use, environmental parameters, point pictures and landscape photos (N, S, W, E), three new types of observations were added with the Copernicus module. First, the land cover at the point reached by the surveyor is registered. This land cover coincides with the standard land cover registered for all LUCAS points only if the surveyor reaches the LUCAS points. Secondly, the extent of the Copernicus land cover (LUCAS land cover classification at level 2) in the four cardinal directions is assessed, up to a maximum distance of 50-m. Moreover, for each cardinal direction the first new land cover observed within a 50-m buffer around the point, is registered. Thirdly, the width of these new land covers are recorded.

In 2018, out of 337.854 LUCAS points, almost 28% were surveyed using the Copernicus protocol, corresponding to 93.091 points. This represents a unique dataset for remote sensing applications. To illustrate this, a case-study utilising the LUCAS Copernicus points for crop monitoring is presented. Based on these results, a discussion will be initiated regarding usefulness, strengths, and weaknesses, of the newly acquired LUCAS Copernicus data. Doing so, this study intends to demonstrate the potential of the LUCAS Copernicus survey for Earth Observation applications, but also to encourage future improvements to the LUCAS Copernicus survey methodology.

There is currently a lack of in situ data for calibration and validation of products derived from the increasingly abundant volume of high resolution satellite imagery such as Landsat and Sentinel-2. Cropland is one of the more difficult classes to automatically recognize from imagery while crop type classification is even more challenging and requires high quality, abundant reference information, which is often not available. The last decade has seen massive technological innovations, in particular the proliferation of mobile applications and open access to very high resolution satellite imagery through applications such as Google Earth and Bing Maps. In addition, crowdsourcing (or outsourcing of tasks to the crowd) and citizen science (or the involvement of citizens in scientific research) have also emerged as effective ways of engaging the public in numerous tasks including data collection, either in situ or online.
A number of tools have been developed in the Center for Earth Observation and Citizen Science at the International Institute for Applied Systems Analysis (IIASA) that have relevance for the development of agricultural EO applications. This presentation will provide an overview of the main online and mobile tools that have been developed including:

• Geo-Wiki: an online application for visual interpretation of very high resolution satellite imagery. Previous crowdsourcing campaigns have included collection of a reference data set for the validation of cropland maps or cropland classes within land cover products and the collection of detailed information on field size, which has led to the production of a global field size map.
• LACO-Wiki: an openly available online land cover validation tool (https://www.laco-wiki.net) that allows users to upload a land cover map, create a sample, interpret the sample using very high resolution satellite imagery (or imagery provided by the user) and calculate an accuracy report. Such a tool can be used to validate cropland maps.
• LACO-Wiki Mobile: an open source smart phone application that allows the sample created in LACO-Wiki to be viewed on the phone and validated in situ. Users are either directed to sample locations or they can classify land cover or land use at any location. The mobile app can also be used for gathering data in situ on cropland or for more specific types of agricultural land use, i.e. different crop types. This tool has been developed through the ESA-funded CrowdVal project in collaboration with CS and GISAT.
• Cropland Capture / Picture Pile: tools for rapid image assessment, either satellite imagery or geo-tagged photographs, both mobile and online. Cropland Capture was used to collect 5 million samples of cropland while Picture Pile, as the next generation tool, will be used to crowdsource oil palm and other crop types in Indonesia in the context of the RESTORE+ project.
• FotoQuest GO: a mobile app where users collect land use and land cover data in situ at LUCAS (Land Use Cover Area frame Survey) sample locations. Campaigns have been run in Austria and across Europe. A visual decision tree guides users in classifying cropland and crop types but automatic crop type detection may be added in the future.

Together these tools have been used to collect data from very high resolution satellite imagery, geo-tagged photos and in situ, which contribute to a reference data set that can be used for land cover, land use and agricultural EO applications. These tools can also be adapted specifically for agricultural data collection, either by experts or the crowd.

Recent launch of Sentinel missions and opening of the Landsat archive to the public gave an outstanding opportunity for researchers to develop new methodologies for monitoring the Earth, particularly in agricultural sector. Whether it is for crop classification, yield prediction, precision agriculture or something else, methodologies often include machine learning algorithms that gain huge popularity in recent years. In order to successfully train, test and put machine learning algorithms into practice, a comprehensive and uniform ground truth data collection is a prerequisite. This is, however, a very tedious task. It is usually done manually, which makes it very sensitive to human errors.
By far the most valuable source of information about the agricultural filed, type of crops, management activities, crop development phases etc. are farmers themselves and their field records. Here, the main problem is to motivate them to share their data. They need to see clear benefits of their participation in such a system and one of the solutions is to give them data about their crops in return.
AgroSense is the official platform of Digital Agriculture of Serbia, which provides support to farmers and agricultural companies in monitoring the growth of crops and planning of the agricultural activities. It was developed by BioSense Institute from Novi Sad and represents an important step in digitization of agriculture and increasing efficiency and competitiveness of Serbian producers. Registration is performed using only the e-mail address and it provides users the access to both web and Android applications. AgroSense web application is designed for visualization and in-depth analysis of data, while the Android application, besides giving instantaneous insight into all data on the field, allows for a quick and easy input of data to the system. Basic services are completely free of charge for the small farmers and they include:
• Digital diary of agricultural activities
• Weather forecast for the location of the parcel
• Satellite-based vegetation indices that describe plant growth, photosynthesis intensity and the availability of water and nutrients
• Overview of soil analysis
• Overview of images of crops
• Information about smart technologies used in agriculture
• Latest information about the occurrence of pests and plant diseases
• Cost analysis
Users can create one or more parcels either by drawing the area on the map or by entering its cadastre information (municipality and parcel ID). Once the parcel is defined, the user can check weather forecast for up to 5 days for that area (temperature, pressure, precipitation, humidity, wind speed and cloudiness), satellite images with vegetation indices and their description, historical information from national meteo stations distributed all over Serbia, start keeping the diary, etc. All relevant information not foreseen by the developers can be documented by using notes.
BioSense held several workshops and info days where participants interested in AgroSense were taught how to start using it. Over 10,000 users have registered for the platform since its launch in October 2017 and they entered information about more than 30,000 parcels. As previously said, one of the key facts about the platform is that it is completely free of charge for all basic users. This way farmers are stimulated to use the platform in everyday work.
AgroSense made a breakthrough in Serbia where farmers’ decisions mainly rely on their previous experience without consideration of capabilities that novel technologies offer. It is a unique place where farmers can get all the information necessary for quality decision making. This is an initial step towards educating farmers about digital technologies and digitization of the agricultural sector in Serbia.
From the scientific point of view, researchers are interested in farmer’s diary of agricultural activities, results of soil analysis carried out and images of crops taken by the farmer, since these can be taken as ground truth data for their research. The diary allows entering information about tillage, fertilisation, planting, plant protection and harvesting. Each has specific form that needs to be filled out (e.g. for planting it is production type, crop, hybrid, plants/ha, depth, row spacing, product price/ha, operational price/ha, etc.), as well as the date of the activity. This richness of information provides unprecedented possibility for developing data-driven decision support systems for agricultural management.
Benefits from this crowdsourcing approach for collecting ground truth data throughout AgroSense platform are undisputed, as it makes field ground truth acquisition obsolete, thus saving time and money. Ground truth data acquired by the researchers can be erroneous, as some crops can be hard to distinguish in particular phenological phases (e.g. wheat and barley). In principle, information recorded by the farmer should be more reliable compared to data acquired by the researcher. However, misconduct can occur, so special algorithms for filtering the database are planned for the future. Detection of outliers, comparison to previous years and researcher’s collected ground truth data, and activity log files may be taken into account when filtering malicious records.
Richness of the data and its comprehensiveness is what makes AgroSense a very useful platform for scientific purposes. By using a standardized form for collecting the data, the uniformity of the input data is preserved. Timeliness is assured by the simple and intuitive GUI which is enabling farmers to save the data on the field by using the mobile app, at the very moment when the activity is performed. As users input the data about the production, the size of the database increases allowing for new and more quality data-driven ML algorithms to be developed, and as the number and quality of modules increases, the more people will find the platform beneficial and start using it regularly. This positive feedback that the platform creates is probably its unique selling-point and is the guarantee of its success in the years to come.
AgroSense is available at www.agrosense.rs.