New Preprint: Enhancing Scale Development: Pseudo Factor Analysis of Language Embedding Similarity Matrices

We build on recent work using Large Language Models (LLMs) in psychometrics to generate pseudo-discrimination parameters. While earlier work focused on pseudo-discrimination at the item-by-construct level, we introduce Pseudo-Factor Analysis to support scale design. It is a data-free, model-based approach to evaluating key aspects of a latent construct’s measurement model, such as dimensionality and relations between factors and indicators. In two studies using Five- and Six-factor personality frameworks, various sentence transformer models, and three encoding methods (atomic, atomic reversed, and macro), pseudo-factor analyses recovered theoretically expected structures. These structures aligned closely with empirical factor structures based on human rating data from prior research. We propose Pseudo-Factor Analysis as a useful method for evaluating and refining items after generation and before trialing. A Shiny app is provided to compute pseudo-factor parameters and related psychometric estimates. [Preprint]