Any place the human species inhabits is inevitably modified by them. One of the first features that appear everywhere, in urban areas as well as in the countryside or deep forests, are roads. Further, roads and streets in general reflect their omnipresent and significant role in our lives through the flow of goods, people, and even culture and information. However, their contribution to the public is highly influenced by their surface. Yet, research on automated road surface classification from remotely sensed data is peculiarly scarce. This work investigates the capacities of chosen convolutional neural networks (fully convolutional network (FCN), U-Net, SegNet, DeepLabv3+) on this task. We find that convolutional neural network (CNN) are capable of distinguishing between compact (asphalt, concrete) and modular (paving stones, tiles) surfaces for both roads and sidewalks on aerial data of spatial resolution of 10 cm. U-Net proved its position as the best-performing model among the tested ones, reaching an overall accuracy of nearly 92%. Furthermore, we explore the influence of adding a near-infrared band to the basic red green blue (RGB) scenes and stress where it should be used and where avoided. Overfitting strategies such as dropout and data augmentation undergo the same examination and clearly show their pros and cons. Convolutional neural networks are also compared to single-pixel based random forests and show indisputable advantage of the context awareness in convolutional neural networks, U-Net reaching almost 25% higher accuracy than random forests. We conclude that convolutional neural networks and U-Net in particular should be considered as suitable approaches for automated semantic segmentation of road surfaces on aerial imagery, while common overfitting strategies should only be used under particular conditions.

Department of Geomatics Faculty of Civil Engineering Czech Technical University Prague Prague Czech Republic

Mundialis GmbH and Co KG Bonn North Rhine Westphalia Germany

See more in PubMed

Alexakis EB, Armenakis C. Improving CNN-based building semantic segmentation using object boundaries. In: Jiang J, Shaker A, Zhang H, editors. ISPRS Congress: Imaging Today, Foreseeing Tomorrow, Commission III, volume 43 of International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Nice, France: Copernicus Gesellschaft mbH; 2022. pp. 41–48.

Assiss JC, Giacomini HC, Ribeiro MC. Road permeability index: evaluating the heterogeneous permeability of roads for wildlife crossing. Ecological Indicators. 2019;99:365–374. doi: 10.1016/j.ecolind.2018.12.012. DOI

Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017;12:2481–2495. doi: 10.1109/TPAMI.2016.2644615. PubMed DOI

Barrington-Leigh C, Millard-Ball A. A century of sprawl in the United States. Proceedings of the National Academy of Sciences of the United States of America. 2015;112:8244–8249. doi: 10.1073/pnas.1504033112. PubMed DOI PMC

Barrington-Leigh C, Millard-Ball A. The world’s user-generated road map is more than 80% complete. PLOS ONE. 2017;12:e0180698. doi: 10.1371/journal.pone.0180698. PubMed DOI PMC

Breiman L. Random forests. Machine Learning. 2001;45(1):5–32. doi: 10.1023/A:1010933404324. DOI

Cao H, Gao Y, Cai W, Xu Z, Li L. Segmentation detection method for complex road cracks collected by UAV based on hc-unet++ Drones. 2023;7(3):189. doi: 10.3390/drones7030189. DOI

Cao X, Zhang K, Jiao L. Csanet: cross-scale axial attention network for road segmentation. Remote Sensing. 2023;15(1):3. doi: 10.3390/rs15010003. DOI

CC-BY-4 Creative commons attribution 4.0 international. 2013. http://creativecommons.org/licenses/by/4.0 http://creativecommons.org/licenses/by/4.0

Center for International Earth Science Information Network–CIESIN–Columbia University and Information Technology Outreach Services–ITOS–University of Georgia Global roads open access data set, version 1 (gROADSv1) 2013. DOI

Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Semantic image segmentation with deep convolutional nets and fully connected CRFs. 2014. https://arxiv.org/abs/1412.7062 https://arxiv.org/abs/1412.7062 PubMed

Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. 2017a. https://arxiv.org/abs/1606.00915 https://arxiv.org/abs/1606.00915 PubMed

Chen LC, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. 2017b. https://arxiv.org/abs/1706.05587 https://arxiv.org/abs/1706.05587

Chen LC, Zhu YK, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, editors. European Conference on Computer Vision (ECCV) Berlin, Germany: Springer; 2018. pp. 833–851.

Chen M, Liu Y. NOx removal from vehicle emissions by functionality surface of asphalt road. Journal of Hazardous Materials. 2010;174:375–379. doi: 10.1016/j.jhazmat.2009.09.062. PubMed DOI

Chollet F. Xception: deep learning with depthwise separable convolutions. Conference on Computer Vision and Pattern Recognition (CVPR); 2017. pp. 1800–1807.

Cuenot F, Fulton L, Staub J. The prospect for modal shifts in passenger transport worldwide and impacts on energy use and CO2. Energy Policy. 2012;41:98–106. doi: 10.1016/j.enpol.2010.07.017. DOI

Dekking F, Kraaikamp C, Lopuhaä H, Meester L. A modern introduction to probability and statistics, understanding why and how. London: Springer-Verlag; 2005.

Dice LR. Measures of the amount of ecologic association between species. Ecology. 1945;3:297–302. doi: 10.2307/1932409. DOI

Ding C, Weng L, Xia M, Lin H. Non-local feature search network for building and road segmentation of remote sensing image. ISPRS International Journal of Geo-Information. 2021;10(4):245. doi: 10.3390/ijgi10040245. DOI

DL-DE DL-DE Zero 2.0 license. 2021. https://www.govdata.de/dl-de/zero-2-0 https://www.govdata.de/dl-de/zero-2-0

Duan Z, Liu J, Ling X, Zhang J, Liu Z. Ernet: a rapid road crack detection method using low-altitude UAV remote sensing images. Remote Sensing. 2024;16(10):1741. doi: 10.3390/rs16101741. DOI

Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Teh YW, Titterington M, editors. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research. Cambridge: Proceedings of Machine Learning Research; 2010. pp. 249–256.

Gunning D, Aha DW. DARPA’s explainable artificial intelligence program. AI Magazine. 2019;2(2):44–58. doi: 10.1609/aimag.v40i2.2850. DOI

Guth J, Wursthorn S, Keller S. Multi-parameter estimation of average speed in road networks using fuzzy control. ISPRS International Journal of Geo-Information. 2020;9(1):55. doi: 10.3390/ijgi9010055. DOI

Haklay M, Weber P. OpenStreetMap: user-generated street maps. IEEE Pervasive Computing. 2008;7(4):12–18. doi: 10.1109/MPRV.2008.80. DOI

He C, He B, Tu M, Wang Y, Qu T, Wang D, Liao M. Fully convolutional networks and a manifold graph embedding-based algorithm for PolSAR image classification. Remote Sensing. 2020;12(9):1467. doi: 10.3390/rs12091467. DOI

He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Piscataway: IEEE; 2016. pp. 770–778.

Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. 2012. https://arxiv.org/abs/1207.0580 https://arxiv.org/abs/1207.0580

Hirose I, Tsunomura M, Shishikura M, Ishii T, Yoshimura Y, Ogawa-Ochiai K, Tsumura N. U-Net-based segmentation of microscopic images of colorants and simplification of labeling in the learning process. Journal of Imaging. 2022;8(7):177. doi: 10.3390/jimaging8070177. PubMed DOI PMC

Hoeser T, Kuenzer C. Object detection and image segmentation with deep learning on earth observation data: a review-part I: evolution and recent trends. Remote Sensing. 2020;10(10):1667. doi: 10.3390/rs12101667. DOI

Hoeser T, Bachofer F, Kuenzer C. Object detection and image segmentation with deep learning on earth observation data: a review-part I: Applications. Remote Sensing. 2020;18(18):3053. doi: 10.3390/rs12183053. DOI

Ibisch PL, Hoffmann MT, Kreft S, Pe’er G, Kati V, Biber-Freudenberger L, DellaSala DA, Vale MM, Hobson PR, Selva N. A global map of roadless areas and their conservation status. Science. 2016;354(6318):1423–1427. doi: 10.1126/science.aaf7166. PubMed DOI

Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. 2017. https://arxiv.org/abs/1502.03167 https://arxiv.org/abs/1502.03167

Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. ArXiv preprint. DOI

Lee D, Kim J, Kim M, Lee H. Intelligent tire sensor-based real-time road surface classification using an artificial neural network. Sensors. 2021;21(9):3233. doi: 10.3390/s21093233. PubMed DOI PMC

Liaw A, Wiener M. Classification and regression by randomforest. R News. 2002;2(3):18–22.

Li C, Ashblock J, White D, Vennapusa P. Permeability and stiffness assessment of paved and unpaved roads with geocomposite drainage layers. Applied Sciences. 2017;7(7):718. doi: 10.3390/app7070718. DOI

Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Piscataway: IEEE; 2015. pp. 6195–6211. PubMed

Masino J, Pinay J, Reischl M, Gauterin F. Road surface prediction from acoustical measurements in the tire cavity using support vector machine. Applied Acoustics. 2017;125(3):41–48. doi: 10.1016/j.apacoust.2017.03.018. DOI

Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine. 1978;8(4):283–298. doi: 10.1016/S0001-2998(78)80014-2. PubMed DOI

MIT The MIT license. 1987. https://opensource.org/license/mit/ https://opensource.org/license/mit/

Mohammadi M. Road classification and condition determination using hyperspectral imagery. In: Shortis M, Wagner W, Hyyppa J, editors. ISPRS Congress, Technical Commission, volume 39 of International Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences. Melbourne, Australia: Copernicus Gesellschaft mbH; 2012. pp. 141–146.

Moretti L, Cantisani G, Carpiecci M, d’Andrea A, del Serrone G, di Mascio O, Loprencipe G. Effect of sampietrini pavers on urban heat islands. International Journal of Environmental Research and Public Health. 2021;18(24):13108. doi: 10.3390/ijerph182413108. PubMed DOI PMC

Nelson A, de Sherbinin A, Pozzi F. Towards development of a high quality public domain global roads database. Data Science Journal. 2006;5(7):223–265. doi: 10.2481/dsj.5.223. DOI

Nolte M, Kister N, Maurer M. Assessment of deep convolutional neural networks for road surface classification. In: Teh YW, Titterington M, editors. International Conference on Intelligent Transportation Systems (ITSC), Volume 21 of IEEE International Conference on Intelligent Transportation Systems-ITSC. Vol. 21. Piscataway: IEEE; 2018. pp. 381–386.

Pedrayes OD, Lema DG, Garcia DF, Usamentiaga R, Alonso A. Evaluation of semantic segmentation methods for land use with spectral imaging using Sentinel-2 and PNOA imagery. Remote Sensing. 2021;13(12):2292. doi: 10.3390/rs13122292. DOI

Pešek O. Convolutional neural networks for road surface classification on aerial imagery. 2024a. https://github.com/pesekon2/road-surface-detection-aerial https://github.com/pesekon2/road-surface-detection-aerial

Pešek O. Road surface aerial photo training dataset. 2024b. DOI

Pešek O, Segal-Rozenhaimer M, Karnieli A. Using convolutional neural networks for cloud detection on VENμS images over multiple land-cover types. Remote Sensing. 2022;14(20):5210. doi: 10.3390/rs14205210. DOI

Piramanayagam S, Saber E, Schwartzkopf W, Koehler FW. Supervised classification of multisensor remotely sensed images using a deep learning framework. Remote Sensing. 2018;12(9):1429. doi: 10.3390/rs10091429. DOI

Ramachandran P, Zoph B, Le QV. Searching for activation functions. 2014. https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941

Razavian AS, Azizpour H, Sullivan J, Carlsson S. CNN features off-the-shelf: an astounding baseline for recognition. Proceeding IEEE Conference on Computer Vision and Pattern Recognition; Piscataway: IEEE; 2014. pp. 512–519.

Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Proceeding International Conference on Medical Image Computing and Computer-Assisted Intervention MICCAI; Cham, Switzerland: Springer; 2015. pp. 234–241.

Sifre L. Rigid-motion scattering for image classification. 2014. PhD thesis, Centre de Mathématiques Appliquées - École Polytechnique, Palaiseau, France.

Silva W, Picado-Santos L, Barroso S, Cabral AE, Stefanutti R. Assessment of interlocking concrete block pavement with by-products and comparison with an asphalt pavement: a review. Applied Sciences. 2023;13(10):5846. doi: 10.3390/app13105846. DOI

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556

Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging. 2015;15(1):178. doi: 10.1186/s12880-015-0068-x. PubMed DOI PMC

Torres DL, Feitosa RQ, Happ PN, La Rosa LEC, Marcato J, Martins J, Bressan PO, Gonçalves WN, Liesenberg V. Applying fully convolutional architectures for semantic segmentation of a single tree species in urban environment on high resolution UAV optical imagery. Sensors. 2021;20(2):563. doi: 10.3390/s20020563. PubMed DOI PMC

Tucker CJ, Gatlin J, Schnieder SR, Kuchinos MA. Monitoring large scale vegetation dynamics in the nile delta and river valley from NOAA-AVHRR data. Conference on Remote Sensing of Arid and Semi-Arid Lands; Cairo, Egypt: 1982. pp. 973–977.

Verburg PH, Ellis EC, Letourneau A. A global assessment of market accessibility and market influence for global environmental change studies. Environmental Research Letters. 2011;6(3):34019. doi: 10.1088/1748-9326/6/3/034019. DOI

Vicente-Saez R, Martinez-Fuentes C. Open science now: a systematic literature review for an integrated definition. Journal of Business Research. 2018;88(4):428–436. doi: 10.1016/j.jbusres.2017.12.043. DOI

Yang JY, Guo JH, Yue HJ, Liu ZH, Hu HF, Li K. CDnet: CNN-based cloud detection for remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing. 2019;8(8):6195–6211. doi: 10.1109/TGRS.2019.2904868. DOI

Ye JC, Sung WK. Understanding geometry of encoder-decoder CNNs. In: Chaudhuri K, Salakhutdinov R, editors. Proceedings of the Machine Learning Research. Norfolk: JMLR-Journal Machine Learning Research; 2019.

Yoo J, Lee C, Lee S, Yoon Y, Lee J, Yum K, Hwang S. Classification of road surfaces based on CNN architecture and tire acoustical signals. Applied Sciences. 2022;12(19):9521. doi: 10.3390/app12199521. DOI

Zijdenbos AP, Dawant BM, Margolin RA, Palmer AC. Morphometric analysis of white-matter lesions in MR images: method and validation. IEEE Transactions on Medical Imaging. 1994;4(4):716–724. doi: 10.1109/42.363096. PubMed DOI