Statistical theory indicates that hierarchical clustering by interviewers or raters needs to be considered to avoid incorrect inferences when performing any analyses including regression, factor analysis (FA) or item response theory (IRT) modelling of binary or ordinal data. We use simulated Positive and Negative Syndrome Scale (PANSS) data to show the consequences (in terms of bias, variance and mean square error) of using an analysis ignoring clustering on confirmatory factor analysis (CFA) estimates. Our investigation includes the performance of different estimators, such as maximum likelihood, weighted least squares and Markov Chain Monte Carlo (MCMC). Our simulation results suggest that ignoring clustering may lead to serious bias of the estimated factor loadings, item thresholds, and corresponding standard errors in CFAs for ordinal item response data typical of that commonly encountered in psychiatric research. In addition, fit indices tend to show a poor fit for the hypothesized structural model. MCMC estimation may be more robust against clustering than maximum likelihood and weighted least squares approaches but further investigation of these issues is warranted in future simulation studies of other datasets. Copyright © 2015 John Wiley & Sons, Ltd.

Department of Kinanthropology Charles University Prague Prague Czech Republic

Department of Psychiatry University of Cambridge Cambridge UK

Hull York Medical School University of York York UK

Mental Health and Addiction Research Group Department of Health Sciences University of York York UK

Social Dimensions of Health Institute and School of Nursing and Midwifery University of Dundee Dundee UK

See more in PubMed

Bentler M.P., Chou C.P. (1987) Practical issues in structural modeling. Sociological Methods & Research, 16(1), 78–117, DOI: 10.1177/0049124187016001004 DOI

Boker S., Neale M., Maes H., Wilde M., Spiegel M., Brick T., Spies J., Estabrook R., Kenny S., Bates T., Mehta P., Fox J. (2011) OpenMx: an open source extended structural equation modeling framework. Psychometrika, 76(2), 306–317, DOI: 10.1007/s11336-010-9200-6 PubMed DOI PMC

Boomsma A., Hoyle R.H., Panter A.T. (2012) The structural equation modeling research report In Hoyle R.H. (ed.) Handbook of Structural Equation Modeling, pp. 341–358, New York, The Guilford Press.

JBrowne M.W., Cudeck R. (1992) Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258, DOI: 10.1177/0049124192021002005 DOI

De Leeuw J., Kreft G.G. (1986) Random coefficient models for multilevel analysis. Journal of Educational Statistics, 11, 57–85, DOI: 10.3102/10769986011001057 DOI

Fox J.P. (2005) Multilevel IRT using dichotomous and polytomous response data. British Journal of Mathematical & Statistical Psychology, 58(1), 145–172, DOI: 10.1348/000711005X38951 PubMed DOI

Fox J., Nie Z., Byrnes J. (2014) sem: Structural Equation Models. R package version 3.1‐5. http://CRAN.R-project.org/package=sem

Gelman A., Hill J. (2007) Data Analysis using Regression and Multilevel/Hierarchical Models, Cambridge, Cambridge University Press.

Goldstein H. (2011) Multilevel Statistical Models, Chichester, John Wiley & Sons.

Goldstein H., Browne W.J. (2005) Multilevel factor analysis models for continuous and discrete data In Maydeu‐Olivares A., McArdle J.J. (eds) Contemporary Psychometrics: A Festschrift for Roderick P. McDonald, pp. 453–475, Mahwah, NJ, Erlbaum.

Grilli L., Rampichini C. (2007) Multilevel factor models for ordinal variables. Structural Equation Modeling, 14(1), 1–25, DOI: 10.1080/10705510709336734 DOI

Hallquist M. (2012) MplusAutomation: Automating Mplus model estimation and interpretation. R package version 0.5‐3. http://CRAN.R-project.org/package=MplusAutomation

Hox J.J. (1993) Factor analysis of multilevel data: gauging the Muthén method In Oud J.H.L., van Blokland‐Vogelesang R.A.W. (eds) Advances in Longitudinal and Multivariate Analysis in the Behavioral Sciences, pp. 141–156, Nijmegen, ITS.

Jöreskog K.G. (1970) A general method for analysis of covariance structures. Biometrika, 57, 239–251, DOI: 10.2307/2334833 DOI

Jöreskog K.G., Sörbom, D. (1989) LISREL 7: A Guide to the Program and Applications, Chicago, IL, SPSS Publications.

Julian M.W. (2001) The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8(3), 325–352, DOI: 10.1207/S15328007SEM0803_1 DOI

Kay S.R., Fiszbein A., Opler L.A. (1987) The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13(2), 261–276, DOI: 10.1093/schbul/13.2.261 PubMed DOI

Kay S.R., Opler L.A., Lindenmayer J.‐P. (1988) Reliability and validity of the positive and negative syndrome scale for schizophrenics. Psychiatry Research, 23(1), 99–110. PubMed

Lindstrom E., Wieselgren I.M., von Knorring L. (1994) Interrater reliability of the Structured Clinical Interview for the Positive and Negative Syndrome Scale for schizophrenia. Acta Psychiatrica Scandinavica, 89(3), 192–195. PubMed

Lunn D.J., Thomas A., Best N., Spiegelhalter D. (2000) WinBUGS – a Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing, 10, 325–337, DOI: 10.1023/a:1008929526011 DOI

McArdle J.J., McDonald R.P. (1984) Some algebraic properties of the Reticular Action Model for moment structures. British Journal of Mathematical & Statistical Psychology, 37, 234–251, DOI: 10.1111/j.2044-8317.1984.tb00802.x PubMed DOI

McDonald R.P. (1994) The bilevel reticular action model for path analysis with latent variables. Sociological Methods & Research, 22, 399–413, DOI: 10.1177/0049124194022003007 DOI

McDonald R.P. (1999) Test Theory: A Unified Treatment, Mahwah, NJ, Erlbaum.

Muthén B. (1984) A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 46, 115–132, DOI: 10.1007/bf02294210 DOI

Muthén B. (1989) Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557–585, DOI: 10.1007/bf02296397 DOI

Muthén B. (1991) Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354, DOI: 10.1111/j.1745-3984.1991.tb00363.x DOI

Muthén B. (1994) Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376–398, DOI: 10.1177/0049124194022003006 DOI

Muthén L., Muthén B. (1998–2013) Mplus: Statistical Analysis with Latent Variables (Version 7.11). Los Angeles, CA, Muthén & Muthén.

Muthén B., Satorra A. (1995) Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316, DOI: 10.2307/271070 DOI

Olsson U.H., Foss T., Troye S.V., Howell R.D. (2000) The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling, 7(4), 557–595, DOI: 10.1207/S15328007SEM0704_3 DOI

Peralta V., Cuesta M.J. (1994) Psychometric properties of the Positive and Negative Syndrome Scale (PANSS) in schizophrenia. Psychiatry Research, 53(1), 31–40, DOI: 0165‐1781(94)90093‐0 [pii] PubMed

Rabe‐Hesketh S., Skrondal A., Pickles A. (2004) Generalized multilevel structural equation modelling. Psychometrika, 69(2), 167–190, DOI: 10.1007/bf02295939 DOI

Rosseel Y. (2012) lavaan: an R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

Rouquette A., Falissard B. (2011) Sample size requirements for the internal validation of psychiatric scales. International Journal of Methods in Psychiatric Research, 20(4), 235–249, DOI: 10.1002/mpr.352 PubMed DOI PMC

Stochl J., Jones P.B., Plaistow J., Reininghaus U., Priebe S., Perez J., Croudace T.J. (2014) Multilevel ordinal factor analysis of the positive and negative syndrome scale (PANSS). International Journal of Methods in Psychiatric Research, 23(1), 25–35, DOI: 10.1002/mpr.1429 PubMed DOI PMC

White L., Harvey P.D., Opler L., Lindenmayer J.P. (1997) Empirical assessment of the factorial structure of clinical symptoms in schizophrenia. A multisite, multimodel evaluation of the factorial structure of the Positive and Negative Syndrome Scale. The PANSS Study Group. Psychopathology, 30(5), 263–274. PubMed