The development of the Conceptual Construct Validity Appraisal Checklist

Abstract

The validation of instruments is integral to sound psychometric assessment practices. Construct validity, as the apex indicator, is established largely based on data-reduction techniques. This bias overlooks the importance of expert-based content validity and alternate approaches to construct validity. The construct definition, object classification, attribute classification, rater identification, scale formation, and enumeration and reporting (C-OAR-SE) framework provides an alternative to traditional approaches to construct validation. However, the theory lacks articulation into an appraisal instrument that can form an integral part of the methodology used for validation studies. There is a need to focus on the conceptual clarity of constructs and its subsequent impact on operationalisation as a prerequisite to data reduction. This study reports on the development of the Conceptual Construct Validity Appraisal Checklist (CCVAC), based on the C-OAR-SE framework. The CCVAC assesses conceptual construct validity as the logical underpinning of instruments. The CCVAC has three sections, assessing theoretical definitions, operational definitions and scoring, respectively. The measure produces subsection and section scores and a global interpretation matrix. The construction followed a rigorous five-step process. The CCVAC was piloted on the Emotional Social Screening Tool for School Readiness (E3SR). The CCVAC performed well and appears non-sensitive to theoretical frameworks. The E3SR achieved a moderate level of construct validity as estimated by the CCVAC. The kappa statistic (0.55) indicated a moderate inter-rater agreement. All ethics principles were adhered to.

Contribution: The CCVAC is theoretically grounded and provides a quantifiable methodology to objectively assess conceptual construct validity. The CCVAC makes the methodology underpinning construction explicit and produces quantifiable outcomes.