Classicism vs. eliminative connectionism: Learning simple artificial grammars with backpropagation networks

One of the central controversies in cognitive science in the last decade is the issue of what kind of mental structure could support complex and systematic behaviour. On one hand, classicists believe that the human mind operates on explicitly structured symbolic representations, where mental representations are characterized by syntactic and semantic structure, and that mental processes operating over those representations are sensitive to their syntactic structure. On the other hand, eliminative connectionists believe the mind is a system composed of simple processing elements (i.e., nodes, units) that resemble biological neurons, and connections that resemble biological synapses between neurons, which can exhibit intelligent behaviour without operating on explicitly structured symbolic representations. In this thesis I closely look into a number of aspects of this controversy by analysing how several neural network models are able to perform an important high-level cognitive task, such as language understanding. I discovered that although a few of these connectionist models can learn some very specific and simple syntactic patterns, they all have serious problems generalizing their knowledge to novel input, especially when this input is formed with more complex (finite-state) grammars. The major reason for this behaviour is the fact that the training regimen employed by each of those neural networks renders the networks incapable of extracting the sequential structure of the input stimuli. With regard to the debate between classicism and eliminative connectionism, I argue that neither classicism, nor eliminative connectionism can explain the entire realm of high-level cognitive processes. Instead, I argue for a paradigm that embodies both classical, traditional symbolic methods and connectionist models.