The performance of deep neural networks and the low costs of computational hardware has made computer vision a popular choice in many robotic systems. An attractive feature of deep-learned methods is their ability to cope with appearance changes caused by day-night cycles and seasonal variations. However, deep learning of neural networks typically relies on large numbers of hand-annotated images, which requires significant effort for data collection and annotation. We present a method that allows autonomous, self-supervised training of a neural network in visual teach-and-repeat (VT&R) tasks, where a mobile robot has to traverse a previously taught path repeatedly. Our method is based on a fusion of two image registration schemes: one based on a Siamese neural network and another on point-feature matching. As the robot traverses the taught paths, it uses the results of feature-based matching to train the neural network, which, in turn, provides coarse registration estimates to the feature matcher. We show that as the neural network gets trained, the accuracy and robustness of the navigation increases, making the robot capable of dealing with significant changes in the environment. This method can significantly reduce the data annotation efforts when designing new robotic systems or introducing robots into new environments. Moreover, the method provides annotated datasets that can be deployed in other navigation systems. To promote the reproducibility of the research presented herein, we provide our datasets, codes and trained models online.

Artificial Intelligence Center Faculty of Electrical Engineering Czech Technical University Prague 166 27 Prague 6 Czech Republic

Department of Computer Science University of Nottingham Jubilee Campus 7301 Wollaton Rd Lenton Nottingham NG8 1BB UK

Zobrazit více v PubMed

DeSouza G.N., Kak A.C. Vision for mobile robot navigation: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2002;24:237–267. doi: 10.1109/34.982903. DOI

Yasuda Y.D., Martins L.E.G., Cappabianco F.A. Autonomous visual navigation for mobile robots: A systematic literature review. ACM Comput. Surv. (CSUR) 2020;53:1–34. doi: 10.1145/3368961. DOI

Sadeghi Amjadi A., Raoufi M., Turgut A.E. A self-adaptive landmark-based aggregation method for robot swarms. Adapt. Behav. 2021 doi: 10.1177/1059712320985543. DOI

Aznar F., Pujol M., Rizo R. Visual navigation for UAV with map references using ConvNets; Proceedings of the Conference of the Spanish Association for Artificial Intelligence; Salamanca, Spain. 14–16 September 2016; Cham, Switzerland: Springer; 2016. pp. 13–22.

Arvin F., Xiong C., Yue S. Colias-φ: An autonomous micro robot for artificial pheromone communication. Int. J. Mech. Eng. Robot. Res. 2015;4:349–353. doi: 10.18178/ijmerr.4.4.349-353. DOI

Crisman J.D., Thorpe C.E. Vision and Navigation. Springer; Cham, Switzerland: 1990. Color vision for road following; pp. 9–24.

Davison A.J., Reid I.D., Molton N.D., Stasse O. MonoSLAM: Real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 2007;29:1052–1067. doi: 10.1109/TPAMI.2007.1049. PubMed DOI

Mur-Artal R., Montiel J.M.M., Tardos J.D. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robot. 2015;31:1147–1163. doi: 10.1109/TRO.2015.2463671. DOI

Taketomi T., Uchiyama H., Ikeda S. Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Trans. Comput. Vis. Appl. 2017;9:1–11. doi: 10.1186/s41074-017-0027-2. DOI

Furgale P., Barfoot T.D. Visual teach and repeat for long-range rover autonomy. J. Field Robot. 2010;27:534–560. doi: 10.1002/rob.20342. DOI

Krajník T., Faigl J., Vonásek V., Košnar K., Kulich M., Přeučil L. Simple yet stable bearing-only navigation. J. Field Robot. 2010;27:511–533. doi: 10.1002/rob.20354. DOI

Lowe D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004;60:91–110. doi: 10.1023/B:VISI.0000029664.99615.94. DOI

Rublee E., Rabaud V., Konolige K., Bradski G. ORB: An efficient alternative to SIFT or SURF; Proceedings of the 2011 International Conference on Computer Vision; Barcelona, Spain. 6–13 November 2011; pp. 2564–2571.

Mukherjee D., Jonathan Wu Q., Wang G. A comparative experimental study of image feature detectors and descriptors. Mach. Vis. Appl. 2015;26:443–466. doi: 10.1007/s00138-015-0679-9. DOI

Valgren C., Lilienthal A.J. SIFT, SURF & seasons: Appearance-based long-term localization in outdoor environments. Robot. Auton. Syst. 2010;58:149–156.

Dayoub F., Cielniak G., Duckett T. Long-term experiments with an adaptive spherical view representation for navigation in changing environments. Robot. Auton. Syst. 2011;59:285–295. doi: 10.1016/j.robot.2011.02.013. DOI

Lowry S., Sünderhauf N., Newman P., Leonard J.J., Cox D., Corke P., Milford M.J. Visual place recognition: A survey. IEEE Trans. Robot. 2015;32:1–19. doi: 10.1109/TRO.2015.2496823. DOI

Paton M., MacTavish K., Warren M., Barfoot T.D. Bridging the appearance gap: Multi-experience localization for long-term visual teach and repeat; Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Daejeon, Korea. 9–14 October 2016; pp. 1918–1925.

Paton M., MacTavish K., Berczi L.P., van Es S.K., Barfoot T.D. Field and Service Robotics. Springer; Cham, Switzerland: 2018. I can see for miles and miles: An extended field test of visual teach and repeat 2.0; pp. 415–431.

Krajník T., Cristóforis P., Kusumam K., Neubert P., Duckett T. Image features for visual teach-and-repeat navigation in changing environments. Robot. Auton. Syst. 2017;88:127–141. doi: 10.1016/j.robot.2016.11.011. DOI

Neubert P., Sünderhauf N., Protzel P. Appearance change prediction for long-term navigation across seasons; Proceedings of the 2013 European Conference on Mobile Robots; Barcelona, Spain. 25–27 September 2013; pp. 198–203.

Zhang N., Warren M., Barfoot T.D. Learning Place-and-Time-Dependent Binary Descriptors for Long-Term Visual Localization; Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA); Brisbane, QLD, Australia. 21–25 May 2018; pp. 828–835. DOI

Sünderhauf N., Shirazi S., Dayoub F., Upcroft B., Milford M. On the performance of convnet features for place recognition; Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Hamburg, Germany. 28 September–2 October 2015; pp. 4297–4304.

Broughton G., Linder P., Rouček T., Vintr T., Krajník T. Robust Image Alignment for Outdoor Teach-and-Repeat Navigation; Proceedings of the 2021 European Conference on Mobile Robots (ECMR); Bonn, Germany. 31 August–3 September 2021; pp. 1–6. DOI

Toft C., Maddern W., Torii A., Hammarstrand L., Stenborg E., Safari D., Okutomi M., Pollefeys M., Sivic J., Pajdla T., et al. Long-term visual localization revisited. IEEE Trans. Pattern Anal. Mach. Intell. 2020 doi: 10.1109/TPAMI.2020.3032010. PubMed DOI

Gridseth M., Barfoot T.D. Keeping an Eye on Things: Deep Learned Features for Long-Term Visual Localization. IEEE Robot. Autom. Lett. 2022;7:1016–1023. doi: 10.1109/LRA.2021.3136867. DOI

Swedish T., Raskar R. Deep visual teach and repeat on path networks; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; Lake City, UT, USA. 18–22 June 2018; pp. 1533–1542.

Rozsypalek Z., Broughton G., Linder P., Roucek T., Kusumam K., Krajnik T. Semi-Supervised Learning for Image Alignment in Teach and Repeat navigation; Proceedings of the Symposium on Applied Computing (SAC) 2022; Brno, Czech Republic. 25–29 April 2022.

Finlayson G.D., Hordley S.D. Color constancy at a pixel. JOSA A. 2001;18:253–264. doi: 10.1364/JOSAA.18.000253. PubMed DOI

Maddern W., Stewart A., McManus C., Upcroft B., Churchill W., Newman P. Illumination invariant imaging: Applications in robust vision-based localisation, mapping and classification for autonomous vehicles; Proceedings of the Visual Place Recognition in Changing Environments Workshop, IEEE International Conference on Robotics and Automation (ICRA); Hong Kong, China. 31 May–5 June 2014; p. 5.

McManus C., Churchill W., Maddern W., Stewart A., Newman P. Shady dealings: Robust, long-term visual localisation using illumination invariance; Proceedings of the IEEE International Conference on Robotics and Automation (ICRA); Hong Kong, China. 31 May–7 June 2014; pp. 901–906. DOI

MacTavish K., Paton M., Barfoot T. Field and Service Robotics (FSR) Springer; Cham, Switzerland: 2015. Beyond a Shadow of a Doubt: Place Recognition with Colour-Constant Images.

Paton M., MacTavish K., Ostafew C., Barfoot T. It’s Not Easy Seeing Green: Lighting-resistant Stereo Visual Teach-and-Repeat Using Color-constant Images; Proceedings of the IEEE International Conference on Robotics and Automation (ICRA); Seattle, WA, USA. 26–30 May 2015.

Dayoub F., Duckett T. An adaptive appearance-based map for long-term topological localization of mobile robots; Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems; Nice, France. 22–26 September 2008; pp. 3364–3369.

Rosen D.M., Mason J., Leonard J.J. Towards lifelong feature-based mapping in semi-static environments; Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA); Stockholm, Sweden. 16–21 May 2016; pp. 1063–1070.

Dymczyk M., Stumm E., Nieto J., Siegwart R., Gilitschenski I. Will It Last? Learning Stable Features for Long-Term Visual Localization; Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV); Stanford, CA, USA. 25–28 October 2016; pp. 572–581. DOI

Berrio J.S., Ward J., Worrall S., Nebot E. Identifying robust landmarks in feature-based maps; Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV); Paris, France. 9–12 June 2019; pp. 1166–1172. DOI

Luthardt S., Willert V., Adamy J. LLama-SLAM: Learning High-Quality Visual Landmarks for Long-Term Mapping and Localization; Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC); Maui, HI, USA. 4–7 November 2018; pp. 2645–2652. DOI

Mühlfellner P., Bürki M., Bosse M., Derendarz W., Philippsen R., Furgale P. Summary maps for lifelong visual localization. J. Field Robot. 2016;33:561–590. doi: 10.1002/rob.21595. DOI

Churchill W.S., Newman P. Experience-based navigation for long-term localisation. Int. J. Robot. Res. 2013;32:1645–1661. doi: 10.1177/0278364913499193. DOI

Linegar C., Churchill W., Newman P. Work smart, not hard: Recalling relevant experiences for vast-scale but time-constrained localisation; Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA); Seattle, WA, USA. 26–30 May 2015; pp. 90–97.

MacTavish K., Paton M., Barfoot T.D. Visual triage: A bag-of-words experience selector for long-term visual route following; Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA); Singapore. 29 May–3 June 2017; pp. 2065–2072. DOI

Neubert P., Sünderhauf N., Protzel P. Superpixel-based appearance change prediction for long-term navigation across seasons. Robot. Auton. Syst. 2014;69:15–27. doi: 10.1016/j.robot.2014.08.005. DOI

Lowry S.M., Milford M.J., Wyeth G.F. Transforming morning to afternoon using linear regression techniques; Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA); Hong Kong, China. 31 May–7 June 2014; pp. 3950–3955.

Krajník T., Vintr T., Molina S., Fentanes J.P., Cielniak G., Mozos O.M., Broughton G., Duckett T. Warped Hypertime Representations for Long-Term Autonomy of Mobile Robots. IEEE Robot. Autom. Lett. 2019;4:3310–3317. doi: 10.1109/LRA.2019.2926682. DOI

Song B., Chen W., Wang J., Wang H. Long-Term Visual Inertial SLAM based on Time Series Map Prediction; Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Macau, China. 3–8 November 2019; pp. 5364–5369. DOI

Halodová L., Dvořráková E., Majer F., Vintr T., Mozos O.M., Dayoub F., Krajník T. Predictive and adaptive maps for long-term visual navigation in changing environments; Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Macau, China. 3–8 November 2019; pp. 7033–7039.

Isola P., Zhu J.Y., Zhou T., Efros A.A. Image-to-Image Translation with Conditional Adversarial Networks; Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Honolulu, HI, USA. 21–26 July 2017.

Porav H., Maddern W., Newman P. Adversarial training for adverse conditions: Robust metric localisation using appearance transfer; Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA); Brisbane, Australia. 21–25 May 2018; pp. 1011–1018.

Cho Y., Jeong J., Shin Y., Kim A. DejavuGAN: Multi-temporal image translation toward long-term robot autonomy; Proceedings of the ICRA Workshop; Brisbane, Australia. 21–25 May 2018; pp. 1–4.

Choi Y., Kim N., Hwang S., Park K., Yoon J.S., An K., Kweon I.S. KAIST Multi-spectral Day/Night Dataset for Autonomous and Assisted Driving. IEEE Trans. Intell. Transp. Syst. (TITS) 2018;19:934–948. doi: 10.1109/TITS.2018.2791533. DOI

Geiger A., Lenz P., Stiller C., Urtasun R. Vision meets robotics: The kitti dataset. Int. J. Robot. Res. 2013;32:1231–1237. doi: 10.1177/0278364913491297. DOI

Carlevaris-Bianco N., Ushani A.K., Eustice R.M. University of Michigan North Campus long-term vision and lidar dataset. Int. J. Robot. Res. 2015 doi: 10.1177/0278364915614638. DOI

Maddern W., Pascoe G., Linegar C., Newman P. 1 year, 1000 km: The Oxford RobotCar dataset. Int. J. Robot. Res. 2017;36:3–15. doi: 10.1177/0278364916679498. DOI

Yan Z., Sun L., Krajník T., Ruichek Y. EU long-term dataset with multiple sensors for autonomous driving; Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Las Vegas, NV, USA. 24 October 2020; pp. 10697–10704.

Calonder M., Lepetit V., Strecha C., Fua P. BRIEF: Binary robust independent elementary features; Proceedings of the Computer Vision ICCV—ECCV 2010, 11th European Conference on Computer Vision; Heraklion, Crete, Greece. 5–11 September 2010.

Neubert P., Protzel P. Local region detector+ CNN based landmarks for practical place recognition in changing environments; Proceedings of the ECMR; Lincoln, UK. 2–4 September 2015; pp. 1–6.

Taisho T., Kanji T. Mining DCNN landmarks for long-term visual SLAM; Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO); Qingdao, China. 3–7 December 2016; pp. 570–576. DOI

Cadena C., Carlone L., Carrillo H., Latif Y., Scaramuzza D., Neira J., Reid I., Leonard J.J. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans. Robot. 2016;32:1309–1332. doi: 10.1109/TRO.2016.2624754. DOI

Kunze L., Hawes N., Duckett T., Hanheide M., Krajník T. Artificial intelligence for long-term robot autonomy: A survey. IEEE Robot. Autom. Lett. 2018;3:4023–4030. doi: 10.1109/LRA.2018.2860628. DOI

Macario Barros A., Michel M., Moline Y., Corre G., Carrel F. A Comprehensive Survey of Visual SLAM Algorithms. Robotics. 2022;11:24. doi: 10.3390/robotics11010024. DOI

Krajník T., Cristóforis P., Nitsche M., Kusumam K., Duckett T. Image features and seasons revisited; Proceedings of the 2015 European Conference on Mobile Robots (ECMR); Lincoln, UK. 2–4 September 2015; pp. 1–7.

Chen Z., Birchfield S.T. Qualitative vision-based path following. IEEE Trans. Robot. Autom. 2009;25:749–754. doi: 10.1109/TRO.2009.2017140. DOI

Royer E., Lhuillier M., Dhome M., Lavest J.M. Monocular vision for mobile robot localization and autonomous navigation. Int. J. Comput. Vis. 2007;74:237–260. doi: 10.1007/s11263-006-0023-y. DOI

Krajník T., Majer F., Halodová L., Vintr T. Navigation without localisation: Reliable teach and repeat based on the convergence theorem; Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Madrid, Spain. 1–5 October 2018; pp. 1657–1664.

De Cristóforis P., Nitsche M., Krajník T., Pire T., Mejail M. Hybrid vision-based navigation for mobile robots in mixed indoor/outdoor environments. Pattern Recognit. Lett. 2015;53:118–128. doi: 10.1016/j.patrec.2014.10.010. DOI

Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. Commun. ACM. 2017;60:84–90. doi: 10.1145/3065386. DOI

Rosten E., Drummond T. European Conference on Computer Vision (ECCV) Springer; Cham, Switzerland: 2006. Machine learning for high-speed corner detection.

Olid D., Fácil J.M., Civera J. Single-View Place Recognition under Seasonal Changes; Proceedings of the PPNIV Workshop at IROS; Madrid, Spain. 1–5 October 2018.

Embedding Weather Simulation in Auto-Labelling Pipelines Improves Vehicle Detection in Adverse Conditions