Model selection in item response theory

The problem of model selection is addressed from a general perspective and solutions are considered within the domain of item response theory (IRT). Selection is conceptualized as including both the evaluation of individual models and the simultaneous comparison of multiple candidates. Traditional tests of goodness of fit can often be regarded as dealing with the former situation, while information criteria can only be applied to the latter. The significance of this last point is pursued in some detail. In terms of optimization, it is shown that information criteria do not provide a means of determining how well their various objective functions are satisfied. This implies that some further criterion is required in order to establish whether the candidates recommended by any information criterion are indeed satisfactory. The need for such a criterion motivates the present work. This approach begins by conceptualizing parametric stochastic models as sets of probability distributions. In any given application the purpose of such a model is to predict the relative frequencies with which an outcome variable takes on its values. This notion of prediction is described in terms of the inclusion of the distribution of the outcome variable in the set of distributions implied by the model: If this is not the case, the model is said to be inaccurate. The concept of accuracy then serves as a basis for selection in IRT. In particular, any IRT model can be represented as a manifold embedded in Euclidean space, and the proximity of any observed distribution to a point on this manifold can be interpreted in terms of the norm of their difference. Describing the geometric properties of sets of candidates provides a means of selection that is not tied to any particular set of observations; this is an important area of further investigation.