Dealing with Semantic Anomalies in a Connectionist Network for World Prediction

Humans are able to recognize a grammatically correct but semantically anomalous sentence. On the task of predicting the range of possible next words in a sentence, given the current word as the input, however, many networks (e.g. Elman, 1990, 1993; Christiansen & Chater, 1994; Hadley et al, 2001) that have been proposed are capable of displaying a certain degree of systematicity, but fail in recognizing anomalous sentences. We believe that humans require both syntactic and semantic information to predict the category of the next word in a sentence. Based on an expansion of Hadley's model (Hadley et al, 2001), we present a competitive network, which employs two subnetworks that discern coarse-grained and fine-grained categories respectively, by being trained via different parameter settings. Hence, one of the sub-networks will have a greater capacity for recognizing the syntactic structure of the preceding words, while the other will have a greater capacity for recognizing the semantic structure of the preceding pattern of words. Also, we employ a mechanism to switch attention between the predictions from the two sub-networks, in order to make the global network more closely approximate human behavior. The results show that the network is capable of exhibiting strong systematici ty, as defined by Hadley (Hadley, 1994a). In addition, it is able to predict in compliance with the semantic constraints implied in the training corpus, and deal with grammatically correct but semantically anomalous sentences. We can conclude that the network has provided a more realistic model for human behavior on the task of predicting the range of possible next words in a sentence. iii