A tangible user interface or TUI connects physical objects and digital interfaces. It is more interactive and interesting for users than a classic graphic user interface. This article presents a descriptive overview of TUI's real-world applications sorted into ten main application areas-teaching of traditional subjects, medicine and psychology, programming, database development, music and arts, modeling of 3D objects, modeling in architecture, literature and storytelling, adjustable TUI solutions, and commercial TUI smart toys. The paper focuses on TUI's technical solutions and a description of technical constructions that influences the applicability of TUIs in the real world. Based on the review, the technical concept was divided into two main approaches: the sensory technical concept and technology based on a computer vision algorithm. The sensory technical concept is processed to use wireless technology, sensors, and feedback possibilities in TUI applications. The image processing approach is processed to a marker and markerless approach for object recognition, the use of cameras, and the use of computer vision platforms for TUI applications.

Faculty of Electrical Engineering and Computer Science VSB Technical University of Ostrava 17 Listopadu 15 708 33 Ostrava Poruba Czech Republic

Zobrazit více v PubMed

Blackwell A.F., Fitzmaurice G., Holmquist L.E., Ishii H., Ullmer B. CHI’07, Proceedings of the Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 28 April–3 May 2007. ACM Press; New York, NY, USA: 2007. Tangible user interfaces in context and theory; pp. 2817–2820.

Ishii H., Mazalek A., Lee J. CHI’01, Proceedings of the Extended Abstracts on Human Factors in Computing Systems, Seattle, WA, USA, 31 March–5 April 2001. ACM Press; New York, NY, USA: 2001. Bottles as a minimal interface to access digital information; pp. 187–188.

Fiebrink R., Morris D., Morris M.R. CHI’09, Proceedings of the 27th International Conference on Human Factors in Computing Systems, Boston, MA, USA, 4–9 April 2009. ACM Press; New York, NY, USA: 2009. Meredith Ringel Morris Dynamic mapping of physical controls for tabletop groupware; pp. 471–480.

Shaer O., Hornecker E. Tangible User Interfaces: Past, Present, and Future Directions. Found. Trends Hum. Comput. Interact. 2009;3:1–137. doi: 10.1561/1100000026. DOI

Do-Lenh S., Jermann P., Cuendet S., Zufferey G., Dillenbourg P. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Proceedings of the European Conference on Technology Enhanced Learning 2010: Sustaining TEL: From Innovation to Learning and Practice, Arcelona, Spain, 28 September–1 October 2010. Volume 6383. Springer; Berlin/Heidelberg, Germany: 2010. Task performance vs. learning outcomes: A study of a tangible user interface in the classroom; pp. 78–92.

Mateu J., Lasala M.J., Alamán X. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Proceedings of the Ubiquitous Computing and Ambient Intelligence. Context-Awareness and Context-Driven Interaction, Carrillo, Costa Rica, 2–6 December 2013. Volume 8276. Springer; Cham, Switzerland: 2013. Tangible interfaces and virtual worlds: A new environment for inclusive education; pp. 119–126.

Lucchi A., Jermann P., Zufferey G., Dillenbourg P. TEI’10, Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied Interaction, Cambridge MA, USA, 24–27 January 2010. Association for Computing Machinery; New York, NY, USA: 2010. An empirical evaluation of touch and tangible interfaces for tabletop displays; pp. 177–184.

Yui T., Hashida T. UIST 2016 Adjunct, Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 16–19 October 2016. Association for Computing Machinery; New York, NY, USA: 2016. Floatio: Floating tangible user interface based on animacy perception; pp. 43–45.

Schneider B., Wallace J., Blikstein P., Pea R. Preparing for future learning with a tangible user interface: The case of neuroscience. IEEE Trans. Learn. Technol. 2013;6:117–129. doi: 10.1109/TLT.2013.15. DOI

Ma J., Sindorf L., Liao I., Frazier J. TEI 2015, Proceedings of the 9th International Conference on Tangible, Embedded, and Embodied Interaction, Stanford, CA, USA, 15–19 January 2015. Association for Computing Machinery; New York, NY, USA: 2015. Using a tangible versus a multi-touch graphical user interface to support data exploration at a museum exhibit; pp. 33–40.

Vaz R.I.F., Fernandes P.O., Veiga A.C.R. Proposal of a Tangible User Interface to Enhance Accessibility in Geological Exhibitions and the Experience of Museum Visitors. Procedia Comput. Sci. 2016;100:832–839. doi: 10.1016/j.procs.2016.09.232. DOI

Kaltenbrunner M., Bencina R. TEI ’07, Proceedings of the 1st International Conference on Tangible and Embedded Interaction, Baton Rouge, Louisiana, 15–17 February 2007. Association for Computing Machinery; New York, NY, USA: 2007. reacTIVision; pp. 69–74.

Schneider B., Sharma K., Cuendet S., Zufferey G., Dillenbourg P., Pea E.R. Using mobile eye-trackers to unpack the perceptual benefits of a tangible user interface for collaborative learning. ACM Trans. Comput. Interact. 2016;23 doi: 10.1145/3012009. DOI

Schneider B., Jermann P., Zufferey G., Dillenbourg P. Benefits of a tangible interface for collaborative learning and interaction. IEEE Trans. Learn. Technol. 2011;4:222–232. doi: 10.1109/TLT.2010.36. DOI

Starcic A.I., Cotic M., Zajc M. Design-based research on the use of a tangible user interface for geometry teaching in an inclusive classroom. Br. J. Educ. Technol. 2013;44:729–744. doi: 10.1111/j.1467-8535.2012.01341.x. DOI

Sorathia K., Servidio R. Learning and Experience: Teaching Tangible Interaction & Edutainment. Procedia Soc. Behav. Sci. 2012;64:265–274. doi: 10.1016/j.sbspro.2012.11.031. DOI

Pulli K., Baksheev A., Kornyakov K., Eruhimov V. Real-time computer vision with OpenCV. Commun. ACM. 2012;55:61–69. doi: 10.1145/2184319.2184337. DOI

Campos P., Pessanha S. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Proceedings of the VMR 2011, Virtual and Mixed Reality-New Trends, Orlando, FL, USA, 9–14 July 2011. Volume 6773 LNCS. Springer; Berlin/Heidelberg, Germany: 2011. Designing augmented reality tangible interfaces for kindergarten children; pp. 12–19.

Girouard A., Solovey E.T., Hirshfield L.M., Ecott S., Shaer O., Jacob R.J.K. TEI’07, Proceedings of the First International Conference on Tangible and Embedded Interaction, Baton Rouge, Louisiana, 15–17 February 2007. Association for Computing Machinery; New York, NY, USA: 2007. Smart Blocks: A tangible mathematical manipulative; pp. 183–186.

Almukadi W., Stephane A.L. ITS’15, Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, Madeira, Portugal, 15–18 November 2015. ACM Press; New York, NY, USA: 2015. BlackBlocks: Tangible interactive system for children to learn 3-letter words and basic math; pp. 421–424.

Patten J., Ishii H., Hines J., Pangaro G. Sensetable: A wireless object tracking platform for tangible user interfaces; Proceedings of the Conference on Human Factors in Computing Systems; Seattle, WA, USA. 31 March–5 April 2001; pp. 253–260.

Reinschlüssel A., Alexandrovsky D., Döring T., Kraft A., Braukmüller M., Janßen T., Reid D., Vallejo E., Bikner-Ahsbahs A., Malaka R. Multimodal Algebra Learning: From Math Manipulatives to Tangible User Interfaces. i-com. 2018;17:201–209. doi: 10.1515/icom-2018-0027. DOI

Gajadur D., Bekaroo G. TangiNet: A Tangible User Interface System for Teaching the Properties of Network Cables; Proceedings of the 2019 Conference on Next Generation Computing Applications (NextComp); Pointe-aux-Piments, Mauritius. 19–21 September 2019; pp. 1–6.

Davis J.U., Wu T.-Y., Shi B., Lu H., Panotopoulou A., Whiting E., Yang X.-D. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM; New York, NY, USA: 2020. TangibleCircuits: An Interactive 3D Printed Circuit Education Tool for People with Visual Impairments; pp. 1–13.

Nathoo A., Bekaroo G., Gangabissoon T., Santokhee A. Using tangible user interfaces for teaching concepts of internet of things. Interact. Technol. Smart Educ. 2020;17:133–158. doi: 10.1108/ITSE-09-2019-0061. DOI

Manshad M.S., Pontelli E., Manshad S.J. ASSETS’11, Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, Dundee Scotland, UK, 24–26 October 2011. Association for Computing Machinery; New York, NY, USA: 2011. MICOO (Multimodal Interactive Cubes for Object Orientation): A tangible user interface for the blind and visually impaired; pp. 261–262.

Billinghurst M., Kato H. Collaborative augmented reality. Commun. ACM. 2002;45:65–70. doi: 10.1145/514236.514265. DOI

McGookin D., Robertson E., Brewster S. Clutching at straws: Using tangible interaction to provide non-visual access to graphs; Proceedings of the Conference on Human Factors in Computing Systems; Atlanta, GA, USA. 10–15 April 2010; pp. 1715–1724.

De La Guía E., Lozano M.D., Penichet V.M.R. Educational games based on distributed and tangible user interfaces to stimulate cognitive abilities in children with ADHD. Br. J. Educ. Technol. 2015;46:664–678. doi: 10.1111/bjet.12165. DOI

Jadan-Guerrero J., Jaen J., Carpio M.A., Guerrero L.A. IDC 2015, Proceedings of the The 14th International Conference on Interaction Design and Children. Association for Computing Machinery; New York, NY, USA: 2015. Kiteracy: A kit of tangible objects to strengthen literacy skills in children with Down syndrome; pp. 315–318.

Haro B.P.M., Santana P.C., Magaña M.A. Developing reading skills in children with Down syndrome through tangible interfaces; Proceedings of the ACM International Conference Proceeding Series; Singapore. 7 August 2012; pp. 28–34.

Martin-Ruiz M.L. Foundations of a Smart Toy Development for the Early Detection of Motoric Impairments at Childhood. Int. J. Pediatr. Res. 2015;1:1–5. doi: 10.23937/2469-5769/1510011. DOI

Rivera D., García A., Alarcos B., Velasco J.R., Ortega J.E., Martínez-Yelmo I. Smart toys designed for detecting developmental delays. Sensors. 2016;16:1953. doi: 10.3390/s16111953. PubMed DOI PMC

Lee K., Jeong D., Schindler R.C., Hlavaty L.E., Gross S.I., Short E.J. Interactive block games for assessing children’s cognitive skills: Design and preliminary evaluation. Front. Pediatr. 2018;6:111. doi: 10.3389/fped.2018.00111. PubMed DOI PMC

Al Mahmud A., Soysa A.I. POMA: A tangible user interface to improve social and cognitive skills of Sri Lankan children with ASD. Int. J. Hum. Comput. Stud. 2020;144:102486. doi: 10.1016/j.ijhcs.2020.102486. DOI

Woodward K., Kanjo E., Brown D.J., Inkster B. Adjunct, Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers. ACM; New York, NY, USA: 2020. TangToys: Smart Toys to Communicate and Improve Children’s Wellbeing; pp. 497–499.

Di Fuccio R., Siano G., De Marco A. The Activity Board 1.0: RFID-NFC WI-FI Multitags Desktop Reader for Education and Rehabilitation Applications; Proceedings of the WorldCIST 2017: Recent Advances in Information Systems and Technologies; Terceira Island, Portugal. 30 March–2 April 2017; pp. 677–689.

Strawhacker A., Bers M.U. “I want my robot to look for food”: Comparing Kindergartner’s programming comprehension using tangible, graphic, and hybrid user interfaces. Int. J. Technol. Des. Educ. 2015;25:293–319. doi: 10.1007/s10798-014-9287-7. DOI

Sullivan A., Bers M.U. Robotics in the early childhood classroom: Learning outcomes from an 8-week robotics curriculum in pre-kindergarten through second grade. Int. J. Technol. Des. Educ. 2016;26:3–20. doi: 10.1007/s10798-015-9304-5. DOI

Sapounidis T., Demetriadis S. Tangible versus graphical user interfaces for robot programming: Exploring cross-age children’s preferences. Pers. Ubiquitous Comput. 2013;17:1775–1786. doi: 10.1007/s00779-013-0641-7. DOI

Wang D., Zhang C., Wang H. IDC 2011, Proceedings of the 10th International Conference on Interaction Design and Children, Michigan, USA, 20–23 June 2011. Association for Computing Machinery; New York, NY, USA: 2011. T-Maze: A tangible programming tool for children; pp. 1–10.

Motoyoshi T., Tetsumura N., Masuta H., Koyanagi K., Oshima T., Kawakami H. Tangible gimmick for programming education using RFID systems. IFAC-PapersOnLine. 2016;49:514–518. doi: 10.1016/j.ifacol.2016.10.608. DOI

Kakehashi S., Motoyoshi T., Koyanagi K., Oshima T., Masuta H., Kawakami H. Improvement of P-CUBE: Algorithm education tool for visually impaired persons; Proceedings of the 2014 IEEE Symposium on Robotic Intelligence in Informationally Structured Space (RiiSS); Orlando, FL, USA. 9–12 December 2014; pp. 1–6.

Rong Z., Chan N.F., Chen T., Zhu K. AsianCHI’20, Proceedings of the 2020 Symposium on Emerging Research from Asia and on Asian Contexts and Cultures. ACM; New York, NY, USA: 2020. CodeRhythm: A Tangible Programming Toolkit for Visually Impaired Students; pp. 57–60.

Nathoo A., Gangabissoon T., Bekaroo G. Exploring the Use of Tangible User Interfaces for Teaching Basic Java Programming Concepts: A Usability Study; Proceedings of the 2019 Conference on Next Generation Computing Applications (NextComp); Pointe-aux-Piments, Mauritius. 19–21 September 2019; pp. 1–5.

Liu L., Wang J., Gong H., Guo J., Wang P., Wang Z., Huang L., Yao C. CHI’20, Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM; New York, NY, USA: 2020. ModBot: A Tangible and Modular Making Toolkit for Children to Create Underwater Robots; pp. 1–8.

Merrad W., Héloir A., Kolski C., Krüger A. RFID-based tangible and touch tabletop for dual reality in crisis management context. J. Multimodal User Interfaces. 2021 doi: 10.1007/s12193-021-00370-2. DOI

Merrill D., Sun E., Kalanithi J. CHI’12, Proceedings of the CHI Conference on Human Factors in Computing Systems, Austin, TX, USA, 5 May–10 May 2012. Association for Computing Machinery; New York, NY, USA: 2012. Sifteo cubes; pp. 1015–1018.

Langner R., Augsburg A., Dachselt R. Cubequery: Tangible interface for creating and manipulating database queries; Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces; Dresden, Germany. 16–19 November 2014; pp. 423–426.

Valdes C., Eastman D., Grote C., Thatte S., Shaer O., Mazalek A., Ullmer B., Konkel M.K. Exploring the design space of gestural interaction with active tokens through user-defined gestures; Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Toronto, ON, Canada. 26 April–1 May 2014; pp. 4107–4116.

Jofre A., Szigeti S., Keller S.T., Dong L.-X., Czarnowski D., Tomé F., Diamond S. A tangible user interface for interactive data visualization; Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering; Markham, ON, Canada. 2–4 November 2015; pp. 244–247.

Villafuerte L., Markova M.S., Jorda S. CHI EA’12, Proceedings of the Extended Abstracts on Human Factors in Computing Systems, Austin Texas, USA, 5 May–10 May 2012. Association for Computing Machinery; New York, NY, USA: 2012. Acquisition of social abilities through musical tangible user interface: Children with autism spectrum condition and the reactable; pp. 745–760.

Xambó A., Hornecker E., Marshall P., Jordà S., Dobbyn C., Laney R. Exploring Social Interaction with a Tangible Music Interface. Interact. Comput. 2016;29:248–270. doi: 10.1093/iwc/iww019. DOI

Waranusast R., Bang-Ngoen A., Thipakorn J. Interactive tangible user interface for music learning; Proceedings of the International Conference Image and Vision Computing; Wellington, New Zealand. 27–29 November 2013; pp. 400–405.

Potidis S., Spyrou T. Spyractable: A tangible user interface modular synthesizer; Proceedings of the International Conference on Human-Computer Interaction; Crete, Greece. 22–27 June 2014; pp. 600–611.

Gohlke K., Hlatky M., De Jong B. Physical construction toys for rapid sketching of Tangible User Interfaces; Proceedings of the 9th International Conference on Tangible, Embedded, and Embodied Interaction; Stanford, CA, USA. 15–19 January 2015; pp. 643–648.

Jacobson A., Panozzo D., Glauser O., Pradalier C., Hilliges O., Sorkine-Hornung O. Tangible and modular input device for character articulation. ACM Trans. Graph. 2014;33:82:1–82:12. doi: 10.1145/2601097.2601112. DOI

Lee J.Y., Seo D.W., Rhee G.W. Tangible authoring of 3D virtual scenes in dynamic augmented reality environment. Comput. Ind. 2011;62:107–119. doi: 10.1016/j.compind.2010.07.003. DOI

Weichel C., Alexander J., Karnik A., Gellersen H. SPATA: Spatio-tangible tools for fabrication-aware design; Proceedings of the 9th International Conference on Tangible, Embedded, and Embodied Interaction; Stanford, CA, USA. 15–19 January 2015; pp. 189–196.

Ishii H., Ullmer B. Tangible bits: Towards seamless interfaces between people, bits and atoms; Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems; Atlanta, GA, USA. 22 March 1997; pp. 234–241.

Ishii H. The tangible user interface and its evolution. Commun. ACM. 2008;51:32–36. doi: 10.1145/1349026.1349034. DOI

Piper B., Ratti C., Ishii H. Illuminating clay: A 3-D tangible interface for landscape analysis; Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Minneapolis, MN, USA. 20–25 April 2002; pp. 355–362.

Ishii H., Ratti C., Piper B., Wang Y., Biderman A., Ben-Joseph E. Bringing clay and sand into digital design-continous tangible user interfaces. BT Technol. J. 2004;22:287–299. doi: 10.1023/B:BTTJ.0000047607.16164.16. DOI

Nasman J., Cutler B. Evaluation of user interaction with daylighting simulation in a tangible user interface. Autom. Constr. 2013;36:117–127. doi: 10.1016/j.autcon.2013.08.018. DOI

Cutler B., Sheng Y., Martin S., Glaser D., Andersen M. Interactive selection of optimal fenestration materials for schematic architectural daylighting design. Autom. Constr. 2008;17:809–823. doi: 10.1016/j.autcon.2008.01.002. DOI

Maquil V., De Sousa L., Leopold U., Tobias E. A geospatial tangible user interface to support stakeholder participation in urban planning; Proceedings of the 2015 1st International Conference on Geographical Information Systems Theory, Applications and Management (GISTAM); Barcelona, Spain. 28–30 April 2015; pp. 1–8.

Ha T., Woo W., Lee Y., Lee J., Ryu J., Choi H., Lee K. ARtalet: Tangible user interface based immersive augmented reality authoring tool for digilog book; Proceedings of the 2010 International Symposium on Ubiquitous Virtual Reality; Gwangju, Korea. 7–10 July 2010; pp. 40–43.

Smith A., Reitsma L., Van Den Hoven E., Kotzé P., Coetzee L. Towards preserving indigenous oral stories using tangible objects; Proceedings of the 2011 2nd International Conference on Culture and Computing, Culture and Computing; Kyoto, Japan. 20–22 October 2011; pp. 86–91.

Wallbaum T., Ananthanarayan S., Borojeni S.S., Heuten W., Boll S. Thematic Workshops’17, Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Mountain View, CA, USA, 23–27 October 2017. ACM Press; New York, NY, USA: 2017. Towards a Tangible Storytelling Kit for Exploring Emotions with Children; pp. 10–16.

Song Y., Yang C., Gai W., Bian Y., Liu J. A new storytelling genre: Combining handicraft elements and storytelling via mixed reality technology. Vis. Comput. 2020;36:2079–2090. doi: 10.1007/s00371-020-01924-3. DOI

Vonach E., Gerstweiler G., Kaufmann H. ACTO: A modular actuated tangible user interface object; Proceedings of the 2014 9th ACM International Conference on Interactive Tabletops and Surfaces (ITS 2014); Dresden, Germany. 16–19 November 2014; pp. 259–268.

Kubicki S., Lepreux S., Kolski C. RFID-driven situation awareness on TangiSense, a table interacting with tangible objects. Pers. Ubiquitous Comput. 2012;16:1079–1094. doi: 10.1007/s00779-011-0442-9. DOI

Zappi P., Farella E., Benini L. Hidden Markov models implementation for tangible interfaces; Proceedings of the Third International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN 2009); Amsterdam, The Netherlands. 22–24 June 2009; pp. 258–263.

Henderson S., Feiner S. Opportunistic tangible user interfaces for augmented reality. IEEE Trans. Vis. Comput. Graph. 2010;16:4–16. doi: 10.1109/TVCG.2009.91. PubMed DOI

Lee J.Y., Kim M.S., Kim J.S., Lee S.M. Tangible user interface of digital products in multi-Displays. Int. J. Adv. Manuf. Technol. 2012;59:1245–1259. doi: 10.1007/s00170-011-3575-0. DOI

Wu F.G., Wang P.C., Tseng W.T., Cheng C.M., Chou Y.C., Sung Y.H. Use pattern-recognition-based technology to explore a new interactive system on smart dining table; Proceedings of the 2013 1st International Conference on Orange Technologies (ICOT); Tainan, Taiwan. 12–16 March 2013; pp. 252–255.

Park D., Choo H. Vibration Based Tangible Tokens for Intuitive Pairing Among Smart Devices; Proceedings of the 4th International Conference on Human Aspects of Information Security, Privacy, and Trust; Toronto, ON, Canada. 17–22 July 2016; pp. 48–56.

Merrill D., Kalanithi J., Maes P. Siftables: Towards sensor network user interfaces; Proceedings of the 1st International Conference on Tangible and Embedded Interaction; Baton Rouge, LA, USA. 15–17 February 2007; pp. 75–78.

Garber L. Tangible user interfaces: Technology you can touch. Computer. 2012;45:15–18. doi: 10.1109/MC.2012.218. DOI

Cooper J.R., Tentzeris M.M. Novel “Smart Cube” Wireless Sensors with Embedded Processing/Communication/Power Core for “Smart Skins” Applications; Proceedings of the 2012 IEEE SENSORS; Taipei, Taiwan. 28–31 October 2012; pp. 1–4.

Cutler B., Nasman J. Advances in Architectural Geometry 2010. Springer; Heidelberg, Germany: 2010. Interpreting Physical Sketches as Architectural Models; pp. 15–35.