Using logistic regression to detect bias when multiple groups are tested

Abstract

When tests are administered to examinees from different language or cultural backgrounds, the detection of biased items, or differential item functioning (DIF), is a crucial step in ensuring that no one group is at a disadvantage. One problem, however, is that most DIF procedures are only applicable to two-group comparisons. When three or more groups are compared, the use of multiple pair-wise comparisons is problematic due to high Type 1 error rates, and it is not clear whether items identified as DIF between two groups also function differentially for other groups. The purpose of this study was to evaluate the viability of using the logistic regression (LR) procedure to simultaneously detect DIF in multiple groups. Two LR estimation procedures, LR1 and LR2, were applied to data simulated for three groups, with 10 per cent of the items simulated as displaying uniform and non-uniform DIF in groups 2 and 3. In addition, the items were simulated to reflect negligible, moderate, and large DIF effect sizes. LR1 was based on multiple pair-wise comparisons and LR2 was based on simultaneous comparisons. Results indicate that both LR procedures could successfully detect DIF when multiple groups are compared with the LR2 as the preferred technique, given the higher detection rate and the lower Type 1 error rate. However, it is important to consider the effect size as well, to determine whether items detected reflect negligible, moderate, or large DIF.