Neural associative memories for language understanding and robot control

Abstract

This work deals with the understanding of sentences using biologically plausible mechanisms, i.e. spiking neural networks, the efficient simulation of them and the integration and application of neural networks on an autonomous agent. An associative structure has been chosen as basic building block for the developed neural networks. The neuron model is a strongly simplified spiking neuron type that allows for implementing an event-driven simulation scheme suitable for high-performance simulations. In contrast to speech recognition where raw audio data is processed, the sentence understanding networks presented here work on the single word level. The presented approaches to language understanding allow to determine the meaning of a spoken sentence while verifying its grammatical correctness at the same time. The network´s most important part is a grammar module representing grammatical structures of the language that is to be understood, and several additional structures to control the flow of information and to validate the incoming word stream against that grammar. The resulting networks show interesting and desired features such as fault tolerance, robust handling of ambiguities on different levels of processing and the ability to extend their vocabulary by learning novel words during performance. The language networks have been integrated into an autonomous agent. The robot is supposed to perform simple actions with fruits lying on a table in front of it, and is controlled by spoken command sentences. A simple action planning module has been implemented using the same associative neural network structure that has been developed for the language understanding network. The resulting neural robot control software shows to be fast enough to control the robot´s actions as well as to understand input sentences in almost real time using standard laptop machines, demonstrating the high performance and efficiency of the implemented neural network solution.