The article presents an AI-based fungi species recognition system for a citizen-science community. The system's real-time identification too - FungiVision - with a mobile application front-end, led to increased public interest in fungi, quadrupling the number of citizens collecting data. FungiVision, deployed with a human-in-the-loop, reaches nearly 93% accuracy. Using the collected data, we developed a novel fine-grained classification dataset - Danish Fungi 2020 (DF20) - with several unique characteristics: species-level labels, a small number of errors, and rich observation metadata. The dataset enables the testing of the ability to improve classification using metadata, e.g., time, location, habitat and substrate, facilitates classifier calibration testing and finally allows the study of the impact of the device settings on the classification performance. The continual flow of labelled data supports improvements of the online recognition system. Finally, we present a novel method for the fungi recognition service, based on a Vision Transformer architecture. Trained on DF20 and exploiting available metadata, it achieves a recognition error that is 46.75% lower than the current system. By providing a stream of labeled data in one direction, and an accuracy increase in the other, the collaboration creates a virtuous cycle helping both communities.

Center for Macroecology Evolution and Climate Biological Institute University of Copenhagen 1165 Copenhagen Denmark

Department of Cybernetics Faculty of Applied Sciences University of West Bohemia 30100 Pilsen Czech Republic

Department of Cybernetics Faculty of Electrical Engineering Czech Technical University Prague 16636 Prague Czech Republic

Global Biodiversity Information Facility 2100 Copenhagen Denmark

Zobrazit více v PubMed

Dickinson J.L., Shirk J., Bonter D., Bonney R., Crain R.L., Martin J., Phillips T., Purcell K. The current state of citizen science as a tool for ecological research and public engagement. Front. Ecol. Environ. 2012;10:291–297. doi: 10.1890/110236. DOI

Jetz W., McGeoch M.A., Guralnick R., Ferrier S., Beck J., Costello M.J., Fernandez M., Geller G.N., Keil P., Merow C., et al. Essential biodiversity variables for mapping and monitoring species populations. Nat. Ecol. Evol. 2019;3:539–551. doi: 10.1038/s41559-019-0826-1. PubMed DOI

Andrew C., Heegaard E., Kirk P.M., Bässler C., Heilmann-Clausen J., Krisai-Greilhuber I., Kuyper T.W., Senn-Irlet B., Büntgen U., Diez J., et al. Big data integration: Pan-European fungal species observations’ assembly for addressing contemporary questions in ecology and global change biology. Fungal Biol. Rev. 2017;31:88–98. doi: 10.1016/j.fbr.2017.01.001. DOI

van Strien A.J., Boomsluiter M., Noordeloos M.E., Verweij R.J.T., Kuyper T.W. Woodland ectomycorrhizal fungi benefit from large-scale reduction in nitrogen deposition in the Netherlands. J. Appl. Ecol. 2018;55:290–298. doi: 10.1111/1365-2664.12944. DOI

Heilmann-Clausen J., Maruyama P.K., Bruun H.H., Dimitrov D., Læssøe T., Frøslev T.G., Dalsgaard B. Citizen science data reveal ecological, historical and evolutionary factors shaping interactions between woody hosts and wood-inhabiting fungi. New Phytol. 2016;212:1072–1082. doi: 10.1111/nph.14194. PubMed DOI

Bayraktarov E., Ehmke G., O’connor J., Burns E.L., Nguyen H.A., McRae L., Possingham H.P., Lindenmayer D.B. Do big unstructured biodiversity data mean more knowledge? Front. Ecol. Evol. 2019;6:239. doi: 10.3389/fevo.2018.00239. DOI

Chandler M., See L., Copas K., Bonde A.M.Z., López B.C., Danielsen F., Legind J.K., Masinde S., Miller-Rushing A.J., Newman G., et al. Contribution of citizen science towards international biodiversity monitoring. Biol. Conserv. 2017;213:280–294. doi: 10.1016/j.biocon.2016.09.004. DOI

Dickinson J., Zuckerberg B., Bonter D.N. Citizen Science as an Ecological Research Tool: Challenges and Benefits. Annu. Rev. Ecol. Evol. Syst. 2010;41:149–172. doi: 10.1146/annurev-ecolsys-102209-144636. DOI

Geldmann J., Heilmann-Clausen J., Holm T.E., Levinsky I., Markussen B., Olsen K., Rahbek C., Tøttrup A.P. What determines spatial bias in citizen science? Exploring four recording schemes with different proficiency requirements. Divers. Distrib. 2016;22:1139–1149. doi: 10.1111/ddi.12477. DOI

iNaturalist. [(accessed on 27 October 2021)]. Available online: http://www.inaturalist.org.

Deng J., Dong W., Socher R., Li L.J., Li K., Fei-Fei L. ImageNet: A large-scale hierarchical image database; Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition; Miami, FL, USA. 20–25 June 2009; pp. 248–255.

Russakovsky O., Deng J., Su H., Krause J., Satheesh S., Ma S., Huang Z., Karpathy A., Khosla A., Bernstein M., et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 2015;115:211–252. doi: 10.1007/s11263-015-0816-y. DOI

Everingham M., Van Gool L., Williams C.K.I., Winn J., Zisserman A. The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vis. 2010;88:303–338. doi: 10.1007/s11263-009-0275-4. DOI

Goëau H., Bonnet P., Joly A. Plant identification in an open-world (LifeCLEF 2016); Proceedings of the CLEF: Conference and Labs of the Evaluation Forum; Evora, Portugal. 5–8 September 2016.

Goeau H., Bonnet P., Joly A. Plant identification based on noisy web data: The amazing performance of deep learning (LifeCLEF 2017); Proceedings of the CLEF: Conference and Labs of the Evaluation Forum; Dublin, Ireland. 11–14 September 2017.

Joly A., Goëau H., Glotin H., Spampinato C., Bonnet P., Vellinga W.P., Planqué R., Rauber A., Palazzo S., Fisher B., et al. LifeCLEF 2015: Multimedia life species identification challenges; Proceedings of the International Conference of the Cross-Language Evaluation Forum for European Languages; Toulouse, France. 8–11 September 2015; Berlin/Heidelberg, Germany: Springer; 2015. pp. 462–483.

Van Horn G., Mac Aodha O., Song Y., Cui Y., Sun C., Shepard A., Adam H., Perona P., Belongie S. The inaturalist species classification and detection dataset; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City; UT, USA. 18–23 June 2018; pp. 8769–8778.

Wah C., Branson S., Welinder P., Perona P., Belongie S. The Caltech-UCSD Birds-200-2011 Dataset. California Institute of Technology; Pasadena, CA, USA: 2011. Technical Report CNS-TR-2011-001.

Nilsback M.E., Zisserman A. Automated Flower Classification over a Large Number of Classes; Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing; Washington, DC, USA. 16–19 December 2008; pp. 722–729.

Gebru T., Hoffman J., Fei-Fei L. Fine-grained recognition in the wild: A multi-task domain adaptation approach; Proceedings of the IEEE International Conference on Computer Vision; Venice, Italy. 22–29 October 2017; pp. 1349–1358.

He K., Fan H., Wu Y., Xie S., Girshick R. Momentum Contrast for Unsupervised Visual Representation Learning; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Long Beach, CA, USA. 16–20 June 2019; pp. 9729–9738.

Oh Song H., Xiang Y., Jegelka S., Savarese S. Deep metric learning via lifted structured feature embedding; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA. 27–30 June 2016; pp. 4004–4012.

Sohn K. Improved deep metric learning with multi-class n-pair loss objective; Proceedings of the 30th International Conference on Neural Information Processing Systems; Barcelona, Spain. 5–10 December 2016; pp. 1857–1865.

Zhai A., Wu H.Y. Classification is a strong baseline for deep metric learning; Proceedings of the British Machine Vision Conference; Cardiff, UK. 9–12 September 2019.

Li D., Hung W.C., Huang J.B., Wang S., Ahuja N., Yang M.H. European Conference on Computer Vision. Springer; Berlin/Heidelberg, Germany: 2016. Unsupervised visual representation learning by graph-based consistent constraints; pp. 678–694.

Lorenz D., Bereska L., Milbich T., Ommer B. Unsupervised part-based disentangling of object shape and appearance; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Long Beach, CA, USA. 16–20 June 2019; pp. 10955–10964.

Wertheimer D., Hariharan B. Few-shot learning with localization in realistic settings; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Long Beach, CA, USA. 16–20 June 2019; pp. 6558–6567.

Sulc M., Matas J. Improving CNN Classifiers by Estimating Test-Time Priors; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops; Seoul, Korea. 27–28 October 2019.

Sulc M., Picek L., Matas J., Jeppesen T., Heilmann-Clausen J. Fungi Recognition: A Practical Use Case; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV); Snowmass Village, CO, USA. 1–5 March 2020; pp. 2305–2313.

Picek L., Šulc M., Matas J., Heilmann-Clausen J., Jeppesen T.S., Læssøe T., Frøslev T. Danish Fungi 2020–Not Just Another Image Recognition Dataset; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV); Waikoloa, HI, USA. 4–8 January 2022; pp. 1525–1535.

Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks; Proceedings of the Advances in Neural Information Processing Systems; Lake Tahoe, NV, USA. 3–6 December 2012; pp. 1097–1105.

Zhang N., Donahue J., Girshick R., Darrell T. Computer Vision—ECCV 2014, Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014. Springer; Berlin/Heidelberg, Germany: 2014. Part-based R-CNNs for fine-grained category detection; pp. 834–849.

Huang S., Xu Z., Tao D., Zhang Y. Part-stacked cnn for fine-grained visual categorization; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA. 27–30 June 2016; pp. 1173–1182.

Feng Z., Fu K., Zhao Q. Learning to focus and discriminate for fine-grained classification; Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP); Taipei, Taiwan. 22–25 September 2019; pp. 415–419.

Xiao T., Xu Y., Yang K., Zhang J., Peng Y., Zhang Z. The application of two-level attention models in deep convolutional neural network for fine-grained image classification; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Boston, MA, USA. 7–12 June 2015; pp. 842–850.

Liu X., Xia T., Wang J., Yang Y., Zhou F., Lin Y. Fully Convolutional Attention Networks for Fine-Grained Recognition. arXiv. 20171603.06765

Lin T.Y., RoyChowdhury A., Maji S. Bilinear cnn models for fine-grained visual recognition; Proceedings of the IEEE International Conference on Computer Vision; Santiago, Chile. 7–13 December 2015; pp. 1449–1457.

Zhang L., Huang S., Liu W., Tao D. Learning a Mixture of Granularity-Specific Experts for Fine-Grained Categorization; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); Seoul, Korea. 27–28 October 2019.

Cui Y., Song Y., Sun C., Howard A., Belongie S. Large scale fine-grained categorization and domain-specific transfer learning; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City; UT, USA. 18–23 June 2018; pp. 4109–4118.

Ngiam J., Peng D., Vasudevan V., Kornblith S., Le Q.V., Pang R. Domain Adaptive Transfer Learning with Specialist Models. arXiv. 20181811.07056

Goëau H., Bonnet P., Joly A. Overview of ExpertLifeCLEF 2018: How far automated identification systems are from the best experts? In Proceedings of the CLEF-Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018;

Goëau H., Bonnet P., Joly A. Overview of LifeCLEF Plant Identification task 2019: Diving into data deficient tropical countries; Proceedings of the CLEF 2019-Conference and Labs of the Evaluation Forum; Lugano, Switzerland. 9–12 September 2019.

Šulc M., Matas J. Fine-grained recognition of plants from images. Plant Methods. 2017;13:115. doi: 10.1186/s13007-017-0265-4. PubMed DOI PMC

Carranza-Rojas J., Goeau H., Bonnet P., Mata-Montero E., Joly A. Going deeper in the automated identification of Herbarium specimens. BMC Evol. Biol. 2017;17:181. doi: 10.1186/s12862-017-1014-z. PubMed DOI PMC

Khosla A., Jayadevaprakash N., Yao B., Fei-Fei L. Novel Dataset for Fine-Grained Image Categorization; Proceedings of the First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition; Colorado Springs, CO, USA. 20–25 June 2011.

Picek L., Durso A., De Castañeda R.R., Bolon I. Overview of SnakeCLEF 2021: Automatic snake species identification with country-level focus; Proceedings of the CEUR Workshop Proceedings; Virtual, Bucharest, Romania. 21–24 September 2021.

Welinder P., Branson S., Mita T., Wah C., Schroff F., Belongie S., Perona P. Caltech-UCSD Birds 200. California Institute of Technology; Pasadena, CA, USA: 2010. Technical Report CNS-TR-2010-001.

Wei X.S., Xie C.W., Wu J., Shen C. Mask-CNN: Localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recognit. 2018;76:704–714. doi: 10.1016/j.patcog.2017.10.002. DOI

Joly A., Goëau H., Kahl S., Deneu B., Servajean M., Cole E., Picek L., De Castaneda R.R., Bolon I., Durso A., et al. International Conference of the Cross-Language Evaluation Forum for European Languages. Springer; Berlin/Heidelberg, Germany: 2020. Overview of LifeCLEF 2020: A system-oriented evaluation of automated species identification and species distribution Prediction; pp. 342–363.

Joly A., Goëau H., Kahl S., Picek L., Lorieul T., Cole E., Deneu B., Servajean M., Durso A., Bolon I., et al. International Conference of the Cross-Language Evaluation Forum for European Languages. Springer; Berlin/Heidelberg, Germany: 2021. Overview of lifeclef 2021: An evaluation of machine-learning based species identification and species distribution prediction; pp. 371–393.

Global Biodiversity Information Facility. 1999. [(accessed on 27 October 2021)]. Available online: https://www.gbif.org.

Robertson T., Belongie S., Adam H., Kaeser-Chen C., Zhang C., Tan K.C., Liu Y., Brulé D., Deltheil C., Loarie S., et al. Training Machines to Identify Species using GBIF-mediated Datasets. Biodivers. Inf. Sci. Stand. 2019;3:e37230. doi: 10.3897/biss.3.37230. DOI

iNaturalist.org. iNaturalist Research-Grade Observations. 2019. [(accessed on 27 October 2021)]. Available online: https://www.gbif.org/dataset/50c9509d-22c7-4a22-a47d-8c48425ef4a7. DOI

Parham J., Stewart C., Crall J., Rubenstein D., Holmberg J., Berger-Wolf T. An Animal Detection Pipeline for Identification; Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV); Lake Tahoe, NV, USA. 12–15 March 2018; pp. 1075–1083.

Danish Mycological Society. 2009. [(accessed on 27 October 2021)]. Available online: https://svampe.databasen.org.

Frøslev T.G., Heilmann-Clausen J., Lange C., Læssøe T., Petersen J.H., Søchting U., Jeppesen T.S., Vesterholt J. Danish Mycological Society, Fungal Records Database. 2021. [(accessed on 27 October 2021)]. Available online: https://www.gbif.org/dataset/84d26682-f762-11e1-a439-00145eb45e9a. DOI

Heilmann-Clausen J., Bruun H.H., Ejrnæs R., Frøslev T.G., Læssøe T., Petersen J.H. How citizen science boosted primary knowledge on fungal biodiversity in Denmark. Biol. Conserv. 2019;237:366–372. doi: 10.1016/j.biocon.2019.07.008. DOI

Tahir M.W., Zaidi N.A., Rao A.A., Blank R., Vellekoop M.J., Lang W. A Fungus Spores Dataset and a Convolutional Neural Network Based Approach for Fungus Detection. IEEE Trans. Nanobiosci. 2018;17:281–290. doi: 10.1109/TNB.2018.2839585. PubMed DOI

Zhang J., Lu S., Wang X., Du X., Ni G., Liu J., Liu L., Liu Y. Automatic identification of fungi in microscopic leucorrhea images. J. Opt. Soc. Am. A. 2017;34:1484–1489. doi: 10.1364/JOSAA.34.001484. PubMed DOI

Zieliński B., Sroka-Oleksiak A., Rymarczyk D., Piekarczyk A., Brzychczy-Włoch M. Deep learning approach to describe and classify fungi microscopic images. PLoS ONE. 2020;15:e0234806. doi: 10.1371/journal.pone.0234806. PubMed DOI PMC

Van De Vooren J., Polder G., Van der Heijden G. Identification of mushroom cultivars using image analysis. Trans. ASAE. 1992;35:347–350. doi: 10.13031/2013.28610. DOI

Fricker M., Boddy L., Bebber D. Biology of the Fungal Cell. Springer; Berlin/Heidelberg, Germany: 2007. Network organisation of mycelial fungi; pp. 309–330.

Heilmann-Clausen J., Frøslev T.G., Læssøe T., Petersen J.H. Danmarks Svampeatlas 2009–2013. Danish Fungal Atlas; Svampetryk, Denmark: 2019.

Occdownload Gbif.Org. Occurrence Download. 2021. [(accessed on 27 October 2021)]. Available online: https://www.gbif.org/occurrence/download/0030066-210914110416597. DOI

Szegedy C., Ioffe S., Vanhoucke V., Alemi A.A. Inception-v4, inception-resnet and the impact of residual connections on learning; Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence; San Francisco, CA, USA. 4–9 February 2017.

Guadarrama S., Silberman N. TensorFlow-Slim: A lightweight library for defining, training and evaluating complex models in TensorFlow. 2016. [(accessed on 27 October 2021)]. Available online: https://github.com/google-research/tf-slim.

Polyak B.T., Juditsky A.B. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 1992;30:838–855. doi: 10.1137/0330046. DOI

Du Plessis M.C., Sugiyama M. Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Netw. 2014;50:110–119. doi: 10.1016/j.neunet.2013.11.010. PubMed DOI

Saerens M., Latinne P., Decaestecker C. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Comput. 2002;14:21–41. doi: 10.1162/089976602753284446. PubMed DOI

Danish Mycological Society, Google Research.Danish Mycological Society Fungi Embedding Model. TensorFlow Hub. [(accessed on 27 October 2021)]. Available online: https://tfhub.dev/google/svampeatlas/vision/embedder/fungi_V1/1. DOI

Hu J., Shen L., Sun G. Squeeze-and-Excitation Networks; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City; UT, USA. 18–23 June 2018; pp. 7132–7141.

Xie S., Girshick R., Dollár P., Tu Z., He K. Aggregated residual transformations for deep neural networks; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Honolulu, HI, USA. 21–26 July 2017; pp. 1492–1500.

Tan M., Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks; Proceedings of the International Conference on Machine Learning; PMLR, Long Beach, CA, USA. 10–15 June 2019; pp. 6105–6114.

Tan M., Le Q.V. Efficientnetv2: Smaller models and faster training. arXiv. 20212104.00298

Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., et al. An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale; Proceedings of the International Conference on Learning Representations; Virtual Conference. 3–7 May 2021.

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention is all you need; Proceedings of the Advances in Neural Information Processing Systems; Long Beach, CA, USA. 4–9 December 2017; pp. 5998–6008.

Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G., Killeen T., Lin Z., Gimelshein N., Antiga L., et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Wallach H., Larochelle H., Beygelzimer A., d’Alché-Buc F., Fox E., Garnett R., editors. Advances in Neural Information Processing Systems 32. Curran Associates, Inc.; New York, NY, USA: 2019. pp. 8024–8035.

Buslaev A., Iglovikov V.I., Khvedchenya E., Parinov A., Druzhinin M., Kalinin A.A. Albumentations: Fast and Flexible Image Augmentations. Information. 2020;11:125. doi: 10.3390/info11020125. DOI

Berg T., Liu J., Lee S.W., Alexander M.L., Jacobs D.W., Belhumeur P.N. Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds; Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition; Columbus, OH, USA. 24–27 June 2014; pp. 2019–2026.

Guo C., Pleiss G., Sun Y., Weinberger K.Q. On calibration of modern neural networks; Proceedings of the International Conference on Machine Learning; PMLR, Sydney, Australia. 6–11 August 2017; pp. 1321–1330.

Vaicenavicius J., Widmann D., Andersson C., Lindsten F., Roll J., Schön T. Evaluating model calibration in classification; Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics; PMLR, Naha, Japan. 16–18 April 2019; pp. 3459–3467.

Olston C., Fiedel N., Gorovoy K., Harmsen J., Lao L., Li F., Rajashekhar V., Ramesh S., Soyke J. Tensorflow-serving: Flexible, high-performance ML serving. arXiv. 20171712.06139

ImageVision. [(accessed on 27 October 2021)]. Available online: https://svampe.databasen.org/en/imagevision.

Danmarks SvampeAtlas—iOS App. [(accessed on 27 October 2021)]. Available online: https://apps.apple.com/us/app/danmarks-svampeatlas/id1467728588?l=da.

Danmarks SvampeAtlas—Android App. [(accessed on 27 October 2021)]. Available online: https://play.google.com/store/apps/details?id=com.noque.svampeatlas&hl=en&gl=US.

Googles ML Kit. [(accessed on 27 October 2021)]. Available online: https://developers.google.com/ml-kit.

Ceccaroni L., Bibby J., Roger E., Flemons P., Michael K., Fagan L., Oliver J.L. Opportunities and risks for citizen science in the age of artificial intelligence. Citiz. Sci. Theory Pract. 2019;4:1–14. doi: 10.5334/cstp.241. DOI

The power of citizen science to advance fungal conservation

An artificial intelligence model to identify snakes from across the world: Opportunities and challenges for global health and herpetology