Research in Artificial Intelligence (AI) has focused mostly on two extremes: either on small improvements in narrow AI domains, or on universal theoretical frameworks which are often uncomputable, or lack practical implementations. In this paper we attempt to follow a big picture view while also providing a particular theory and its implementation to present a novel, purposely simple, and interpretable hierarchical architecture. This architecture incorporates the unsupervised learning of a model of the environment, learning the influence of one's own actions, model-based reinforcement learning, hierarchical planning, and symbolic/sub-symbolic integration in general. The learned model is stored in the form of hierarchical representations which are increasingly more abstract, but can retain details when needed. We demonstrate the universality of the architecture by testing it on a series of diverse environments ranging from audio/visual compression to discrete and continuous action spaces, to learning disentangled representations.

GoodAI Research s r o Karolinská Prague Czech Republic

See more in PubMed

Hutter M. Universal Artificial Intelligence Sequential Decisions Based on Algorithmic Probability. Springer; 2010.

Wissner-Gross AD, Freer CE. Causal Entropic Forces. Physical Review Letters. 2013;110(16). 10.1103/PhysRevLett.110.168702 PubMed DOI

Schmidhuber J. PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem. Frontiers in Psychology. 2013;4:313 10.3389/fpsyg.2013.00313 PubMed DOI PMC

Wang P. From NARS to a Thinking Machine. In: Proceedings of the 2007 Conference on Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms; 2007. p. 75–93. Available from: http://www.cis.temple.edu/~pwang/.

And BRS, And KRT, Schmidhuber J. Growing Recursive Self-Improvers. In: Artificial General Intelligence—9th International Conference, AGI 2016, New York, NY, USA, July 16-19, 2016, Proceedings. vol. 7716; 2016. p. 1–11. Available from: 10.1007/978-3-319-41649-6. DOI

Franz A. Artificial general intelligence through recursive data compression and grounded reasoning: a position paper. Goethe University Frankfurt; 2015. Available from: http://arxiv.org/abs/1506.04366%5Cnhttp://www.arxiv.org/pdf/1506.04366.pdf.

Hart DA, Goertzel B. OpenCog: A Software Framework for Integrative Artificial General Intelligence. In: AGI; 2008.

Carlson A, Betteridge J, Kisiel B. Toward an Architecture for Never-Ending Language Learning. In: Proceedings of the Conference on Artificial Intelligence (AAAI); 2010. p. 1306–1313. Available from: http://www.aaai.org/ocs/index.php/aaai/aaai10/paper/download/1879/2201%5Cnhttp://www.ncbi.nlm.nih.gov/pubmed/21259302.

Nivel E. Ikon Flux 2.0. Technical Report. 2007;.

Bach J. The MicroPsi Agent Architecture. Proceedings of ICCM5 International Conference on Cognitive Modeling Bamberg Germany. 2003;1(1):15–20.

Franklin S, Patterson FG. The LIDA architecture: Adding new modes of learning to an intelligent, autonomous, software agent. Integrated Design and Process Technology. 2006; p. 1–8.

Kotseruba I, Tsotsos JK. 40 Years of Cognitive Architectures Core Cognitive Abilities and Practical Applications. arXiv preprint arXiv:161008602. 2017;.

Mikolov T, Joulin A, Baroni M. A Roadmap towards Machine Intelligence. arXiv preprint arXiv:151108130. 2015; p. 1–36.

Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building Machines That Learn and Think Like People. arXiv preprint arXiv:160400289. 2016;. PubMed

Bengio Y. The Consciousness Prior. arXiv preprint arXiv:170908568. 2017;abs/1709.0.

Nivel E, Thórisson KR, Steunebrink BR, Dindo H, Pezzulo G, Rodriguez M, et al. Bounded Recursive Self-Improvement. arXiv preprint arXiv:13126764. 2013;(December 2013).

Hay N, Stark M, Schlegel A, Wendelken C, Park D, Purdy E, et al. Behavior is Everything-Towards Representing Concepts with Sensorimotor Contingencies. Vicarious; 2018. Available from: www.aaai.org.

Blouw P, Solodkin E, Thagard P, Eliasmith C. Concepts as Semantic Pointers: A Framework and Computational Model. Cognitive Science. 2016;40(5):1128–1162. 10.1111/cogs.12265 PubMed DOI

Wiskott L, Sejnowski TJ. Slow Feature Analysis: Unsupervised Learning of Invariances. Neural Computation. 2002;770(4):715–770. 10.1162/089976602317318938 PubMed DOI

Machery E, Werning M, Stewart T, Eliasmith C. Compositionality and Biologically Plausible Models In: W Hinzen and E Machery and M Werning, editor. Oxford Handbook of Compositionality. Oxford University Press; 2009. Available from: http://compneuro.uwaterloo.ca/files/publications/stewart.2012.pdf.

Shastri L, Ajjanagadde V. From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony. University of Pennsylvania; 1990. Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.543&rep=rep1&type=pdf.

Marblestone A, Wayne G, Kording K. Towards an integration of deep learning and neuroscience. arXiv preprint arXiv:160603813. 2016. PubMed PMC

Hassabis D, Kumaran D, Summerfield C, Botvinick M. Neuroscience-Inspired Artificial Intelligence. Neuron. 2017;95:245–258. 10.1016/j.neuron.2017.06.011 PubMed DOI

Hawkins J, George D. Hierarchical Temporal Memory Concepts, Theory, and Terminology. Numenta; 2006. Available from: http://www-edlab.cs.umass.edu/cs691jj/hawkins-and-george-2006.pdf.

Lillicrap TP, Cownden D, Tweed DB, Akerman CJ. Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:14110247. 2014;.

Lázaro-Gredilla M, Liu Y, Phoenix DS, George D. Hierarchical compositional feature learning. arXiv preprint arXiv:161102252. 2016; p. 1–18.

Eisenreich B, Akaishi R, Hayden B. Control without controllers: Towards a distributed neuroscience of executive control. doiorg. 2016; p. 077685. PubMed PMC

Yang Qiang, Pan SJ. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering. 2010;22(10):1345–1359. 10.1109/TKDE.2009.191 DOI

Hochreiter S, Urgen Schmidhuber J. Long Short-Term Memory. Neural Computation. 1997;9(8):1735–1780. 10.1162/neco.1997.9.8.1735 PubMed DOI

Santoro A, Bartunov S, Botvinick M, Wierstra D, Lillicrap T. One-shot Learning with Memory-Augmented Neural Networks. arXiv preprint arXiv:160506065. 2016;.

Santoro A, Raposo D, Barrett DGT, Malinowski M, Pascanu R, Battaglia P, et al. A simple neural network module for relational reasoning. arXiv preprint arXiv:170601427. 2017;.

He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:151203385. 2015;abs/1512.0.

Sabour S, Frosst N, Hinton GE, Toronto GB. Dynamic Routing Between Capsules. arXiv preprint arXiv:171009829. 2017;.

Liu Y, Lou X, Laan C, George D, Lehrach W, Kansky K, et al. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science. 2017;358(6368):eaag2612 10.1126/science.aag2612 PubMed DOI

Jo J, Bengio Y. Measuring the tendency of CNNs to Learn Surface Statistical Regularities. arXiv preprint arXiv:171111561. 2017;.

Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, et al. Intriguing properties of neural networks. arXiv preprint arXiv:13126199. 2014;.

Su J, Vargas DV, Kouichi S. One pixel attack for fooling deep neural networks. arXiv preprint arXiv:171008864. 2017;.

Roy A. A theory of the brain—the brain uses both distributed and localist (symbolic) representation. In: The 2011 International Joint Conference on Neural Networks. IEEE; 2011. p. 215–221. Available from: http://ieeexplore.ieee.org/document/6033224/.

Bach J. Representations for a Complex World: Combining Distributed and Localist Representations for Learning and Planning. University of Osnabrück; 2005. Available from: http://cognitive-ai.com/publications/assets/BachBiomimeticsBook05Feb09.pdf.

Feldman J. The neural binding problem(s). Cognitive neurodynamics. 2013;7(1):1–11. 10.1007/s11571-012-9219-8 PubMed DOI PMC

Deisenroth MP, Neumann G, Peters J. A Survey on Policy Search for Robotics. Foundations and Trends R in Robotics. 2011;2:1–2. 10.1561/2300000021 DOI

Higgins I, Pal A, Rusu A, Matthey L, Burgess C, Pritzel A, et al. DARLA: Improving Zero-Shot Transfer in Reinforcement Learning. arXiv preprint arXiv:170708475. 2018;.

Ha D, Schmidhuber J. World Models. CoRR. 2018;abs/1803.1.

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–533. 10.1038/nature14236 PubMed DOI

Blundell C, Uria B, Pritzel A, Li Y, Ruderman A, Leibo JZ, et al. Model-Free Episodic Control. arXiv preprint arXiv:160604460. 2016; p. 1–12.

Besold TR, D’ A, Garcez A, Bader S, Bowman H, Domingos P, et al. Neural-Symbolic Learning and Reasoning Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. arXiv preprint arXiv:171103902. 2017;abs/1711.0.

Choo X, Eliasmith C. General Instruction Following in a Large-Scale Biologically Plausible Brain Model. In: 35th Annual Conference of the Cognitive Science Society. Cognitive Science Society; 2013. p. 322–327.

Canziani A, Culurciello E. CortexNet: a Generic Network Family for Robust Visual Temporal Representations. arXiv preprint arXiv:170602735. 2017;abs/1706.0(1).

Rasmus A, Valpola H, Honkala M, Berglund M, Raiko T. Semi-Supervised Learning with Ladder Networks. arXiv preprint arXiv:150702672. 2015;.

Piekniewski F, Laurent P, Petre C, Richert M, Fisher D, Hylton TL. Unsupervised Learning from Continuous Video in a Scalable Predictive Recurrent Network. arXiv preprint arXiv:160706854. 2016;.

Rinkus GJ. Sparsey™: event recognition via deep hierarchical sparse distributed codes. Frontiers in computational neuroscience. 2014;8:160 10.3389/fncom.2014.00160 PubMed DOI PMC

O’reilly RC, Wyatte DR, Rohrlich J. Deep Predictive Learning: A Comprehensive Model of Three Visual Streams. arXiv preprint arXiv:170904654. 2017;.

Qiu J, Huang G, Lee TS. A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction. arXiv preprint arXiv:190109002. 2019;.

Laukien E, Crowder R, Byrne F. Feynman Machine: The Universal Dynamical Systems Computer. arXiv preprint arXiv:160903971. 2016;.

Hawkins J, Ahmad S. Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex. Frontiers in Neural Circuits. 2016;10 10.3389/fncir.2016.00023 PubMed DOI PMC

Friston K. Hierarchical Models in the Brain. Citation: Friston K PLoS Comput Biol. 2008;4 (1110). PubMed PMC

Socolar JES. Nonlinear Dynamical Systems In: Complex Systems Science in Biomedicine. Boston, MA: Springer US; 2006. p. 115–140. Available from: http://link.springer.com/10.1007/978-0-387-33532-2_3. DOI

Franz A. On Hierarchical Compression and Power Laws in Nature. In: International Conference on Artificial General Intelligence; 2017. p. 77–86. Available from: https://occam.com.ua/app/uploads/2017/08/AGI17_Arthur_Franz_hierarchical_compression_final.pdf.

Lin HW, Tegmark M, Rolnick D. Why does deep and cheap learning work so well? arXiv preprint arXiv:160808225 2017;.

Lin HW, Tegmark M. Criticality in Formal Languages and Statistical Physics. arXiv preprint arXiv:160606737. 2016.

Oliver N, Garg A, Horvitz E. Layered representations for learning and inferring office activity from multiple sensory channels. Computer Vision and Image Understanding. 2004;96:163–180. 10.1016/j.cviu.2004.02.004 DOI

Fine S. The Hierarchical Hidden Markov Model: Analysis and Applications. Machine Learning. 1998;32(1):41–62. 10.1023/A:1007469218079 DOI

Baum LE, Petrie T. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics. 1966;37(6):1554–1563. 10.1214/aoms/1177699147 DOI

Richert M, Fisher D, Piekniewski F, Izhikevich EM, Hylton TL. Fundamental principles of cortical computation: unsupervised learning with prediction, compression and feedback. arXiv preprint arXiv:160806277. 2016;.

Hawkins J, Ahmad S, Cui Y. Why Does the Neocortex Have Layers and Columns, A Theory of Learning the 3D Structure of the World. bioRxiv. 2017; p. 0–15.

Adams RA, Shipp S, Friston KJ. Predictions not commands: active inference in the motor system. Brain Structure and Function. 2013;218(3):611–643. 10.1007/s00429-012-0475-5 PubMed DOI PMC

Higgins I, Matthey L, Glorot X, Pal A, Uria B, Blundell C, et al. Early Visual Concept Learning with Unsupervised Deep Learning. arXiv preprint arXiv:160605579. 2016;.

Hinton GE, McClelland JL, Rumelhart DE. Chapter 3-Distributed representations. Rumelhart DE, McClelland JL, PDP Research Group C, editors. Cambridge, MA, USA: MIT Press; 1986. Available from: http://dl.acm.org/citation.cfm?id=104279.104287.

Indyk P, Motwd R. Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality. In: Proceedings of the thirtieth annual ACM symposium on Theory of computing; 1998. p. 604–613. Available from: http://www.cs.princeton.edu/courses/archive/spr04/cos598B/bib/IndykM-curse.pdf.

Thomas V, Bengio E, Fedus W, Pondard J, Beaudoin P, Larochelle H, et al. Disentangling the independently controllable factors of variation by interacting with the world. In: NIPS 2017 Workshop; 2017. Available from: http://arxiv.org/abs/1802.09484.

Spratling MW. A review of predictive coding algorithms. Brain and Cognition. 2017;112:92–97. 10.1016/j.bandc.2015.11.003 PubMed DOI

Narendra KS, Driollet OA, Feiler M, George K. Adaptive control using multiple models, switching and tuning. International Journal of Adaptive Control and Signal Processing. 2003;17(2):87–102. 10.1002/acs.740 DOI

GoodAI. Brain Simulator; 2017. Available from: https://www.goodai.com/brain-simulator.

GoodAI. TorchSim; 2019. Available from: https://github.com/GoodAI/torchsim.

Laukien E, Crowder R, Byrne F. Feynman Machine: A Novel Neural Architecture for Cortical And Machine Intelligence. In: The AAAI 2017 Spring Symposium on Science of Intelligence: Computational Principles of Natural and Artificial Intelligence; 2017. Available from: https://aaai.org/ocs/index.php/SSS/SSS17/paper/viewFile/15362/14605.

ogma ai. video of a bird; 2019. Available from: https://github.com/ogmacorp/OgmaNeoDemos/tree/master/resources.

GoodAI. Video generated by the Expert; 2019. Available from: http://bit.ly/2um5zyc.

Schwartz-Ziv R, Tishby N. Opening the Black Box of Deep Neural Networks via Information. arXiv preprint arXiv:170300810. 2017;.

GoodAI. Original audio file with labels; 2019. Available from: http://bit.ly/2HxdTUA.

GoodAI. Audio generated by one Expert without context; 2019. Available from: http://bit.ly/2W7OXpO.

GoodAI. Audio generated by a hierarchy of 3 Experts; 2019. Available from: http://bit.ly/2FrnFWg.

GoodAI. Illustrative video of the inference; 2019. Available from: http://bit.ly/2CvXnQv.

Laukien E. Original experiment with car; 2017. Available from: https://bit.ly/2XVZmXF.

GoodAI. Autonomous navigation of the agent on the race track; 2019. Available from: http://bit.ly/2OgkVO5.

Dileep G. How brain might work. PhD Thesis. 1987;30(6):541–550.

Bastos AM, Usrey WM, Adams RA, Mangun GR, Fries P, Friston KJ. Canonical Microcircuits for Predictive Coding. 2012. PubMed PMC

Richert M, Fisher D, Piekniewski F, Izhikevich EM, Hylton TL. Fundamental principles of cortical computation: unsupervised learning with prediction, compression and feedback. 2016;.

Ponte Costa R, Assael YM, Shillingford B, Vogels TP. Cortical microcircuits as gated-recurrent neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc.; 2017. p. 271–282. Available from: https://arxiv.org/pdf/1711.02448.pdf.

Yao K, Cohn T, Vylomova K, Duh K, Dyer C. Depth-Gated LSTM. CoRR. 2015; p. 1–5.

Lee TS, Mumford D. Hierarchical Bayesian inference in the visual cortex. J Opt Soc Am A. 2003;20:1434–1448. 10.1364/JOSAA.20.001434 PubMed DOI

Hwang J, Kim J, Ahmadi A, Choi M, Tani J. Predictive coding-based deep dynamic neural network for visuomotor learning. 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob). 2017.

Wayne G, Hung CC, Amos D, Mirza M, Ahuja A, Grabska-Barwinska A, et al. Unsupervised Predictive Memory in a Goal-Directed Agent. 2018;.

Rosa M, Feyereisl J, Collective TG. A Framework for Searching for General Artificial Intelligence. GoodAI; 2016. Available from: http://arxiv.org/abs/1611.00685.

Tan R, Terno DR, Thompson J, Vedral V, Gu M. Towards Quantifying Complexity with Quantum Mechanics. arXiv preprint arXiv:14046255. 2014;.

Brodu N. Reconstruction of Epsilon-Machines in Predictive Frameworks and Decisional States. Advances in Complex Systems. 2011;14(05):761–794. 10.1142/S0219525911003347 DOI

Kullback S, Leibler RA. On Information and Sufficiency. The Annals of Mathematical Statistics. 2007;22(1):79–86. 10.1214/aoms/1177729694 DOI

Sutton RS, Barto AG. Sutton and Barto Book: Reinforcement Learning: An Introduction. IEEE Transactions on Neural Networks. 1988;16:285–286.

Whitehead SD, Lin LJ. Reinforcement learning of non-Markov decision processes. Artificial Intelligence. 1995;73(1-2):271–306. 10.1016/0004-3702(94)00012-P DOI

Friston KJ, Daunizeau J, Kiebel SJ. Reinforcement Learning or Active Inference? PLoS ONE. 2009;4(7). 10.1371/journal.pone.0006421 PubMed DOI PMC

Georgievski I, Aiello M. An Overview of Hierarchical Task Network Planning. arXiv preprint arXiv:14037426. 2014;.

Pezzulo G, Rigoli F, Friston KJ. Hierarchical Active Inference: A Theory of Motivated Control. Trends in Cognitive Sciences. 2018. 10.1016/j.tics.2018.01.009 PubMed DOI PMC

Nagabandi A, Kahn G, Fearing RS, Levine S. Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning. arXiv preprint arXiv:170802596. 2017;.

Kautz H, Mcallester D, Selman B. Encoding Plans in Propositional Logic. In: Proceedings ofthe Fifth International Conference on Principles of Knowledge Representation and Reasoning; 1996. p. 374–384. Available from: http://www.cs.cornell.edu/selman/papers/pdf/96.kr.plan.pdf.

Fikes RE, Nhsson NJ. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence. 1971;2(3-4):189–208. 10.1016/0004-3702(71)90010-5 DOI

Ghallab M, Nau DS, Traverso P. Automated planning and acting. Cambridge University Press; 2016. Available from: http://projects.laas.fr/planning/.

Dietterich TG. Hierarchical Reinforcement Learning with the MAXQ Value FUnction Decomposition. arXiv preprint arXiv:cs/9905014. 1999;cs.LG/9905.

Kulkarni TD, Narasimhan KR, Saeedi A, Tenenbaum JB . Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. arXiv preprint arXiv:160406057. 2016;.

Bacon PL, Harb J, Precup D. The Option-Critic Architecture. School of Computer Science McGill University; 2016. Available from: http://arxiv.org/abs/1609.05140.

Hengst B. Generating Hierarchical Structure in Reinforcement Learning from State Variables; 2000. p. 533–543. Available from: http://link.springer.com/10.1007/3-540-44533-1_54. DOI

Harpur GF, Prager RW. Development of low entropy coding in a recurrent network. Network: Computation in Neural Systems. 1996;7:277–284. 10.1088/0954-898X_7_2_007 PubMed DOI