What, where, and when? Mechanisms of learning biological motion representations

Abstract

The understanding and recognition of human actions is one of the major challenges for technical systems aiming at visual behavior analysis. Evidences from psychophysical and neurophysiological studies provide indications on the feature characteristics and neural processing principles involved in the perception of biological motion sequences. Modeling efforts from the field of computational neuroscience complement these empirical findings by proposing potential functional mechanisms and learning schemes enabling the establishment and recognition of biological motion representations and show how such principles can be transferred to technical domains. First, results of psychophysical investigations are presented that demonstrate significant increases in the human recognition performance for motion (sub-) sequences containing highly articulated poses, which co-occur with local extrema in the motion energy and the extension of a body. Such key poses thus qualify as candidates to establish biological motion representations. Second, based on these findings, a neural model for the learning of biological motion representations is presented. The model combines hierarchical feedforward and feedback processing along the ventral (form; what) and dorsal (motion; where) pathways with an unsupervised Hebbian learning mechanism for the learning of prototypical form and motion representations. More specifically, gated learning in the form pathway realizes the selective learning of highly articulated postures. Sequence selective representations are established using temporal association learning driven by motion and form input. The proposed model shows how the unsupervised learning of key poses can form the basis for the establishment of biological motion representations and gives a potential explanation for empirically observed phenomena, such as implied motion perception. Third, as a transfer to technical application scenarios, a real-time biologically inspired action recognition system is presented which automatically selects key poses in action sequences and employs a deep convolutional neural network (DCNN) to learn class-specific pose representations. The network is mapped onto a neuromorphic platform, enabling the real-time (~1000 fps) and energy-efficient (~70 mW) assignment of key poses to action classes. Last, it is shown how an associative learning scheme similar to the one applied in the neural model for the learning of biological motion representations can be used for the learning of visual category and subcategory representations. Here, instar learning is used to learn representations of visual categories, while outstar learning on the other hand is applied to establish representations of the expected input distribution. The category specific pattern is propagated back to the preceding stage where a residual signal reflecting the difference to the current input signal is derived. This difference is emphasized by modulation of the input with the residual signal and a subsequent normalization. If the difference is large enough, a new subcategory representation is established.