Thinking like a naturalist: enhancing computer vision of citizen science images by harnessing contextual data

Terry, J. Christopher D. ORCID: https://orcid.org/0000-0002-0626-9938; Roy, Helen E. ORCID: https://orcid.org/0000-0001-6050-679X; August, Tom A. ORCID: https://orcid.org/0000-0003-1116-3385. 2020 Thinking like a naturalist: enhancing computer vision of citizen science images by harnessing contextual data. Methods in Ecology and Evolution, 11 (2). 303-315. https://doi.org/10.1111/2041-210X.13335

Before downloading, please read NORA policies.

Preview

Text N526777JA.pdf - Published Version Available under License Creative Commons Attribution 4.0. Download (1MB) | Preview

Abstract/Summary

1. The accurate identification of species in images submitted by citizen scientists is currently a bottleneck for many data uses. Machine learning tools offer the potential to provide rapid, objective and scalable species identification for the benefit of many aspects of ecological science. Currently, most approaches only make use of image pixel data for classification. However, an experienced naturalist would also use a wide variety of contextual information such as the location and date of recording. 2. Here, we examine the automated identification of ladybird (Coccinellidae) records from the British Isles submitted to the UK Ladybird Survey, a volunteer‐led mass participation recording scheme. Each image is associated with metadata; a date, location and recorder ID, which can be cross‐referenced with other data sources to determine local weather at the time of recording, habitat types and the experience of the observer. We built multi‐input neural network models that synthesize metadata and images to identify records to species level. 3. We show that machine learning models can effectively harness contextual information to improve the interpretation of images. Against an image‐only baseline of 48.2%, we observe a 9.1 percentage‐point improvement in top‐1 accuracy with a multi‐input model compared to only a 3.6% increase when using an ensemble of image and metadata models. This suggests that contextual data are being used to interpret an image, beyond just providing a prior expectation. We show that our neural network models appear to be utilizing similar pieces of evidence as human naturalists to make identifications. 4. Metadata is a key tool for human naturalists. We show it can also be harnessed by computer vision systems. Contextualization offers considerable extra information, particularly for challenging species, even within small and relatively homogeneous areas such as the British Isles. Although complex relationships between disparate sources of information can be profitably interpreted by simple neural network architectures, there is likely considerable room for further progress. Contextualizing images has the potential to lead to a step change in the accuracy of automated identification tools, with considerable benefits for large‐scale verification of submitted records.

Item Type:	Publication - Article
Digital Object Identifier (DOI):	https://doi.org/10.1111/2041-210X.13335
UKCEH and CEH Sections/Science Areas:	Biodiversity (Science Area 2017-)
ISSN:	2041-210X
Additional Information. Not used in RCUK Gateway to Research.:	Open Access paper - full text available via Official URL link.
Additional Keywords:	citizen science, computer vision, convolutional neural network, ladybird, machine learning, metadata, naturalists, species identification
NORA Subject Terms:	Ecology and Environment Data and Information
Date made live:	10 Feb 2020 16:23 +0 (UTC)
URI:	https://nora.nerc.ac.uk/id/eprint/526777

Actions (login required)

View Item

Document Downloads

More statistics for this item...