Automated analysis of small and optically variable plant organs, such as grain spikes, is highly demanded in quantitative plant science and breeding. Previous works primarily focused on the detection of prominently visible spikes emerging on the top of the grain plants growing in field conditions. However, accurate and automated analysis of all fully and partially visible spikes in greenhouse images renders a more challenging task, which was rarely addressed in the past. A particular difficulty for image analysis is represented by leaf-covered, occluded but also matured spikes of bushy crop cultivars that can hardly be differentiated from the remaining plant biomass. To address the challenge of automated analysis of arbitrary spike phenotypes in different grain crops and optical setups, here, we performed a comparative investigation of six neural network methods for pattern detection and segmentation in RGB images, including five deep and one shallow neural network. Our experimental results demonstrate that advanced deep learning methods show superior performance, achieving over 90% accuracy by detection and segmentation of spikes in wheat, barley and rye images. However, spike detection in new crop phenotypes can be performed more accurately than segmentation. Furthermore, the detection and segmentation of matured, partially visible and occluded spikes, for which phenotypes substantially deviate from the training set of regular spikes, still represent a challenge to neural network models trained on a limited set of a few hundreds of manually labeled ground truth images. Limitations and further potential improvements of the presented algorithmic frameworks for spike image analysis are discussed. Besides theoretical and experimental investigations, we provide a GUI-based tool (SpikeApp), which shows the application of pre-trained neural networks to fully automate spike detection, segmentation and phenotyping in images of greenhouse-grown plants.

Leibniz Institute of Plant Genetics and Crop Plant Research 06466 Gatersleben Germany

Plant Sciences Core Facility CEITEC Central European Institute of Technology Masaryk University 60200 Brno Czech Republic

PSI spol s r o 66424 Drasov Czech Republic

See more in PubMed

Guo Z., Zhao Y., Röder M.S., Reif J.C., Ganal M.W., Chen D., Schnurbusch T. Manipulation and Prediction of Spike Morphology Traits for the Improvement of Grain Yield in Wheat. Sci. Rep. 2018;8:1–10. doi: 10.1038/s41598-018-31977-3. PubMed DOI PMC

Minervini M., Scharr H., Tsaftaris S.A. Image Analysis: The New Bottleneck in Plant Phenotyping. IEEE Signal Proc. Mag. 2015;32:126–131. doi: 10.1109/MSP.2015.2405111. DOI

Hasan M.M., Chopin J.P., Laga H., Miklavcic S.J. Detection and Analysis of Wheat Spikes Using Convolutional Neural Networks. Plant Methods. 2018;14:100. doi: 10.1186/s13007-018-0366-8. PubMed DOI PMC

Tan C., Zhang P., Zhang Y., Zhou X., Wang Z., Du Y., Mao W., Li W., Wang D., Guo W. Rapid Recognition of Field-grown Wheat Spikes Based on a Superpixel Segmentation Algorithm Using Digital Images. Front. Plant Sci. 2020;11:259. doi: 10.3389/fpls.2020.00259. PubMed DOI PMC

Alharbi N., Zhou J., Wang W. Automatic Counting of Wheat Spikes From Wheat Growth Images; Proceedings of the ICPRAM; Funchal, Portugal. 16–18 January 2018; pp. 346–355.

Costa C., Schurr U., Loreto F., Menesatti P., Carpentier S. Plant Phenotyping Research Trends, a Science Mapping Approach. Front. Plant Sci. 2019;9:1933. doi: 10.3389/fpls.2018.01933. PubMed DOI PMC

Pieruschka R., Schurr U. Plant Phenotyping: Past, Present, and Future. Plant Phenomics. 2019;2019:7507131. doi: 10.34133/2019/7507131. PubMed DOI PMC

Grillo O., Blangiforti S., Venora G. Wheat Landraces Identification through Glumes Image Analysis. Comput. Electron. Agric. 2017;141:223–231. doi: 10.1016/j.compag.2017.07.024. DOI

Bi K., Jiang P., Li L., Shi B., Wang C. Non-destructive Measurement of Wheat Spike Characteristics Based on Morphological Image Processing. Trans. Chin. Soc. Agric. Eng. 2010;26:212–216.

Misra T., Arora A., Marwaha S., Chinnusamy V., Rao A.R., Jain R., Sahoo R.N., Ray M., Kumar S., Raju D. Spikesegnet—A Deep Learning Approach Utilizing Encoder-decoder Network with Hourglass for Spike Segmentation and Counting in Wheat Plant from Visual Imaging. Plant Methods. 2020;16:1–20. doi: 10.1186/s13007-020-00582-9. PubMed DOI PMC

Pound M.P., Atkinson J.A., Wells D.M., Pridmore T.P., French A.P. Deep Learning For Multi-task Plant Phenotyping; Proceedings of the IEEE International Conference on Computer Vision Workshops; Venice, Italy. 22–29 October 2017.

Qiongyan L., Cai J., Berger B., Okamoto M., Miklavcic S.J. Detecting Spikes of Wheat Plants Using Neural Networks with Laws Texture Energy. Plant Methods. 2017;13:83. doi: 10.1186/s13007-017-0231-1. PubMed DOI PMC

Narisetti N., Neumann K., Röder M.S., Gladilin E. Automated Spike Detection In Diverse European Wheat Plants Using Textural Features And The Frangi Filter in 2D Greenhouse Images. Front. Plant Sci. 2020;11:666. doi: 10.3389/fpls.2020.00666. PubMed DOI PMC

Krizhevsky A., Sutskever I., Hinton G.E. Imagenet Classification With Deep Convolutional Neural Networks. Adv. Neural Inf. Process. Syst. 2012;25:1097–1105. doi: 10.1145/3065386. DOI

Everingham M., Williams C.K. The Pascal Visual Object Classes Challenge 2007 (Voc2007) Results. [(accessed on 8 October 2021)]. Available online: http://host.robots.ox.ac.uk/pascal/VOC/voc2007/

Deng J., Dong W., Socher R., Li L.J., Li K., Fei-Fei L. Imagenet: A Large-scale Hierarchical Image Database; Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition; Miami, FL, USA. 20–25 June 2009; pp. 248–255.

Tzutalin. LabelImg. Free Software: MIT License. 2015. [(accessed on 1 December 2019)]. Available online: https://github.com/tzutalin/labelImg.

Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C.Y., Berg A.C. European Conference on Computer Vision. Springer; Berlin/Heidelberg, Germany: 2016. SSD: Single Shot Multibox Detector; pp. 21–37.

Uijlings J.R., Van De Sande K.E., Gevers T., Smeulders A.W. Selective Search For Object Recognition. Int. J. Comput. Vis. 2013;104:154–171. doi: 10.1007/s11263-013-0620-5. DOI

Everingham M., Van Gool L., Williams C.K., Winn J., Zisserman A. The Pascal Visual Object Classes (Voc) Challenge. Int. J. Comput. Vis. 2010;88:303–338. doi: 10.1007/s11263-009-0275-4. DOI

Ren S., He K., Girshick R., Sun J. Faster R-cnn: Towards Real-time Object Detection With Region Proposal Networks. Adv. Neural Inf. Process. Syst. 2015;28:91–99. doi: 10.1109/TPAMI.2016.2577031. PubMed DOI

Redmon J., Farhadi A. Yolov3: An Incremental Improvement. arXiv. 20181804.02767

Bochkovskiy A., Wang C.Y., Liao H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv. 20202004.10934

He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA. 27–30 June 2016; pp. 770–778.

Chang T., Kuo C.C. Texture Analysis And Classification With Tree-structured Wavelet Transform. IEEE Trans. Image Process. 1993;2:429–441. doi: 10.1109/83.242353. PubMed DOI

Christodoulou C.I., Pattichis C.S., Pantziaris M., Nicolaides A. Texture-based Classification of Atherosclerotic Carotid Plaques. IEEE Trans. Med. Imaging. 2003;22:902–912. doi: 10.1109/TMI.2003.815066. PubMed DOI

Ronneberger O., Fischer P., Brox T. U-net: Convolutional Networks For Biomedical Image Segmentation; Proceedings of the International Conference on Medical image computing and Computer-Assisted Intervention; Munich, Germany. 5–9 October 2015; Berlin/Heidelberg, Germany: Springer; 2015. pp. 234–241.

Chen L.C., Zhu Y., Papandreou G., Schroff F., Adam H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation; Proceedings of the ECCV; Munich, Germany. 8–14 September 2018.

Holschneider M., Kronland-Martinet R., Morlet J., Tchamitchian P. Wavelets. Springer; Berlin/Heidelberg, Germany: 1990. A Real-time Algorithm for Signal Analysis with the Help of the Wavelet Transform; pp. 286–297.

Chen L.C., Papandreou G., Kokkinos I., Murphy K., Yuille A.L. Semantic Image Segmentation With Deep Convolutional Nets And Fully Connected CRFs. arXiv. 20141412.7062 PubMed

Huang J., Rathod V., Sun C., Zhu M., Korattikara A., Fathi A., Fischer I., Wojna Z., Song Y., Guadarrama S. Speed/Accuracy Trade-offs For Modern Convolutional Object Detectors; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Honolulu, HI, USA. 21–26 July 2017; pp. 7310–7311.

Kingma D.P., Ba J. Adam: A Method For Stochastic Optimization. arXiv. 20141412.6980

Salehi S.S.M., Erdogmus D., Gholipour A. Tversky loss Function for Image Segmentation using 3D Fully Convolutional Deep Networks; Proceedings of the International Workshop on Machine Learning in Medical Imaging; Quebec City, QC, Canada. 10 September 2017; pp. 379–387.

Azad R., Asadi-Aghbolaghi M., Fathy M., Escalera S. Attention deeplabv3+: Multi-level context attention mechanism for skin lesion segmentation; Proceedings of the European Conference on Computer Vision; Glasgow, UK. 23–28 August 2020; pp. 251–266.

High-Throughput Spike Detection in Greenhouse Cultivated Grain Crops with Attention Mechanisms-Based Deep Learning Models