Abstract
The Montreal Cognitive Assessment (MoCA) is widely used to screen for cognitive impairment and has shown a good capacity to identify cognitive impairment. However, its psychometric properties have not been sufficiently studied in the South African context. Therefore, this study aimed to investigate (1) the influence of demographic variables (age, years of education, and gender) on total MoCA scores; (2) the internal consistency reliability of the test and (3) the discriminant validity of the total MoCA score. This study analysed secondary quantitative data, utilising a cross-sectional, between-subjects design. All participants completed the English MoCA version 8.1. The control sample (n = 89) included healthy South African adults who speak English as a second or third language and who have been educated in public schools. The clinical sample (n = 83) included patients with human immunodeficiency virus (HIV) and a comorbid disorder, either psychiatric (n = 70) or neurocognitive (n = 13). Total MoCA scores were significantly correlated with years of education (p < 0.001) and age (p = 0.007) but not gender. Cronbach’s alpha was 0.64 revealing moderate internal consistency. The total MoCA score was not a significant predictor of diagnostic status, indicating poor discriminant validity of the MoCA in this sample. The MoCA appears not to be a useful screening or diagnostic tool in samples with similar characteristics.
Keywords: HAD; HAND; HIV; Montreal Cognitive Assessment; psychometrics; screening.
Introduction
The Montreal Cognitive Assessment (MoCA) is a brief screening tool developed to detect mild cognitive impairment (MCI), which can be a precursor to dementia (Nasreddine et al., 2005). Subsequently, the MoCA has been widely used in various socioeconomic, cultural and linguistic contexts, including Portugal (Freitas, Simoes, Alves, & Santana, 2012), Italy (Milanini et al., 2016), North America (Rossetti, Lacritz, Cullum, & Weiner, 2011), Brazil (Cecato, Martinelli, Izbicki, Yassuda, & Aprahamian, 2016), China (Zhang et al., 2016) and South Africa (Robbins et al., 2013). The MoCA has been translated into nearly 100 languages (https://www.mocatest.org/faq/), which has promoted its widespread use. Beyond its original scope, the potential of the MoCA for detecting cognitive impairment has also been explored in a range of medical conditions associated with neuropsychological symptoms, including human immunodeficiency virus (HIV; Joska et al., 2016; Robbins et al., 2013).
While the MoCA has garnered international recognition as a valid brief cognitive screening test, the data from the studies conducted in South Africa have been contradictory and inconclusive in terms of its utility, specifically for identifying HIV-related neuropsychological impairment. This may be because of methodological variations and limitations, such as the absence of comparisons between clinical and control samples (Beath, Asmal, Van den Heuvel, & Seedat, 2018; Joska et al., 2016; Rademeyer & Joubert, 2016), and small sample sizes (Hakkers et al., 2018; Rademeyer & Joubert, 2016; Robbins et al., 2013), which limited these studies’ generalisability and comparability. Consequently, our study aims to address some of these gaps and contribute to the limited body of literature on the MoCA in South Africa by exploring the influence of age, years of education and gender, on total MoCA score, and internal consistency, in a sample of South African adults that speak English as a second or third language and have been educated in public schools. Moreover, it investigates the MoCA’s discriminant validity in terms of its capacity to identify cognitive impairment in a sample of patients with dual diagnosis, namely HIV plus a psychiatric or neurocognitive condition, compared to a healthy control sample. By investigating the psychometric properties of the MoCA, this study aims to determine the degree to which this tool can be interpreted consistently across different samples and contexts (i.e. reliable) and the extent to which it accurately measures what it was designed to measure (i.e. valid; De Souza, Costa-Alexandre, & De Brito-Guirardello, 2017).
Several studies have indicated a consistent and significant correlation between demographic variables, such as age and years of education, with performance on the MoCA, whereby younger and highly educated samples have higher scores (Elkana, Tal, Oren, Soffer, 2020; Malek-Ahmadi et al., 2015; Pinto et al., 2018). In contrast, the effect of gender is more ambiguous in the literature; some studies report significant differences in performance on the MoCA between males and females (Kaya et al., 2014; Lu et al., 2011), whereas others do not (Robbins et al., 2013; Santangelo et al., 2015). The impact of demographic and contextual variables on neuropsychological test performance is particularly relevant to consider in South Africa with its culturally and linguistically diverse population.
The impact of language on test performance has been extensively researched (Ferrett et al., 2014; Kisser et al., 2012; Watts & Shuttleworth-Edwards, 2016) and highlights the biases that arise when non-native English speakers are assessed in English and compared against norms of first-language English speakers. This is an issue of particular relevance to South Africa, where English is the sixth most common home language (8.1%) and the second most prevalent language spoken outside of South African households at around 17% (Statista, 2018). This linguistic landscape accentuates the importance of exploring the psychometric performance of tests for non-native English speaker samples who are being assessed in this language, which is a frequent practice in cognitive testing given that most tools are available in English (Laher & Cockcroft, 2013). Moreover, South Africa also has high socioeconomic and educational inequalities, largely a product of apartheid (Gallo, 2020), which, along with cultural and linguistic diversity, introduces significant biases in cognitive and neuropsychological testing (Ng et al., 2018), especially for those educated in under-resourced (typically public) schools (Cockcroft, Alloway, Copello, & Milligan, 2015; Shuttleworth-Edwards et al., 2004; Watts & Shuttleworth-Edwards, 2016). Thus, our study aimed to reduce linguistic biases by selecting healthy participants who were non-native English speakers and attended public schooling and by selecting a clinical sample with similar characteristics.
Biases in neuropsychological testing have a major impact on diagnostic work because they increase the risk of Type II errors, where patients’ performances are incorrectly judged as pathological (Meyer, Boscardin, Kwasa, & Price, 2013). In addition, the use of brief screenings, such as the MoCA, in diverse populations is particularly precarious because the clinician relies on the interpretation of a single score to make judgements on the cognitive status of patients (Wong et al., 2015). This is even more challenging when the patient has more than one diagnosis associated with cognitive deficit. Such challenges are common when assessing people living with HIV because its impact on cognition is variable, which is aggravated in cases with other comorbid conditions (Jonsson et al., 2013). Therefore, clinicians are often faced with the need of conducting brief, valid and reliable assessments to identify cognitive symptoms in order to accurately diagnose and manage patients. However, this need is not met if the tests are not valid and/or reliable and is not interpreted considering the relevant demographic variables and appropriate norms.
Exploring the validity of the MoCA in samples with HIV is particularly relevant in South Africa given its high prevalence at around 12% of the total population (UNAIDS, 2019). Investigating the utility of this screening test in samples with dual diagnosis (HIV and psychiatric/neurocognitive disorders [NCD]) responds to a gap in the literature and to a need for empirical studies to guide clinicians in the selection and use of cognitive screening tools.
Methods
Participants
Sample and sampling
Both the clinical and control samples were formed by means of purposive sampling (Etikan, Musa, & Alkassim, 2016). The criteria for inclusion for the overall sample are adults over the age of 21 years, who have English as a second or third language, did not abuse illicit drugs or alcohol, and who provided formal consent. The participants from the control group had no history of any motor, psychiatric or cognitive symptoms. Patients’ medical files with incomplete and/or contradictory data (such as age, years of education, and/or date of administration) were excluded from data analysis, which had a negative impact on the sample size. Patients were further excluded if they had a comorbidity that was not psychiatric or neurocognitive in nature (n = 16), for example, patients with other neurological conditions, such as meningitis, or metabolic disorders, such as diabetes. Those who had ≤6 years of formal education were also excluded because of the small size of the sample (n = 3). The final clinical sample included participants who had a dual clinical diagnosis, classified into two groups: (1) participants with HIV and comorbid psychiatric disorder, consisting of mood disorder (including bipolar mood disorder and major depressive disorder), psychosis (including schizophrenia), or mood and psychosis disorders (henceforth referred to as the ‘MP’ [mood and psychosis] group) and (2) HIV participants with comorbid NCD, including HIV-associated dementia (HAD) and mild neurocognitive disorder (MND), with or without an additional psychiatric disorder (henceforth referred to as the ‘NCD’ group). The rationale behind this classification is to determine whether the level of severity of cognitive impairment, where NCD is more severe than psychiatric disorder, influences MoCA performance and/or the ability of MoCA to discriminate between these two clinical groups. The final sample included 172 participants in total. The descriptive characteristics of the sample for each of the study variables are summarised in Table 1.
| TABLE 1: Descriptive statistics characterising sample demographics and total Montreal Cognitive Assessment scores. |
Research design
This study utilised a quantitative, non-experimental ex-post facto design, with no manipulation of independent variables or random assignment into groups (Ary, Jacobs, Razavieh, & Ary, 2010). It has a cross-sectional, between-subjects design (Ary et al., 2010).
Instruments
Demographic Questionnaire
Used to obtain demographic data for the control and clinical groups, including age (years), years of education, gender (female; male), handedness, and clinical diagnosis (for patients). This questionnaire also included items exploring language experience whereby participants self-reported their home language, list languages in order of dominance, list languages used daily, and the language/s educated in years. This was done in order to exclude the participants who identified English as the home and primary language.
Montreal Cognitive Assessment
All participants completed the same MoCA English version 8.1. The MoCA can be administered in approximately 10 to 15 min, and measures several domains of neurocognitive function out of a maximum score of 30 points, where higher scores indicate better cognitive function (Nasreddine et al., 2005). Visuospatial abilities are measured using a three-dimensional cube copy drawing (one point) and a clock drawing task with a specified time (three points). Multiple aspects of executive functioning are measured using the alternating trail-making task (one point), two-item verbal abstraction task (two points) and phonemic fluency task (one point). The phonemic fluency task generally assesses aspects of executive functioning as well as language ability. Furthermore, the MoCA measures short-term memory through a five-item delayed word recall task (five points), and attention/concentration and working memory are assessed through a digit span forwards and backwards task (two points), a tapping test (one point), and serial seven subtraction task (three points). Language ability is assessed via a sentence repetition task (two points), a three-item animal naming task (lion, camel, rhinoceros; three points), as well as the aforementioned phonemic fluency task. Finally, the level of conscious awareness of temporal and spatial orientation is assessed using the orientation task, whereby the participant is asked to name the current date, month, year, day of the week, place location, and city (six points). Following the manual’s scoring instructions, an additional point was added to the total score for participants with less than 12 years of formal education (Nasreddine et al., 2005).
Procedure
The present study conducted secondary analyses of MoCA data. For the clinical sample, the MoCA test score data was obtained retrospectively from the patients’ medical files at Luthando Neuropsychiatric Clinic, Chris Hani Baragwanath Hospital, that have prior authorisation for use for research purposes. The MoCAs were administered as a screening test by trained psychiatric registrars under the supervision of a consultant psychiatrist and/or neuropsychiatrist. The control sample data were collected prospectively for a study in which the principal aim was to identify and describe the neurocognitive profiles of individuals with Huntington Disease-Like 2 (HDL2) by means of comparisons to a matched unaffected control sample, in which the MoCA was administered as part of a large battery of neurocognitive tests (for a more detailed description, see Ferreira-Correia, Anderson, Cockcroft, & Krause, 2020). The MoCA examines a wide variety of cognitive domains including memory, language, executive functioning, arithmetic ability, visuospatial skills, and orientation.
Data analysis
Statistical analyses were conducted using the IBM Statistical Package for the Social Sciences (SPSS) version 26.0 for Windows (version 16.0), with the significance level set at 0.05. Descriptive analyses of study variables, including age, gender, years of education and total MoCA score, were conducted to characterise the sample. The distribution of continuous data (age, years of education, and total MoCA score) for the control group was checked using histograms and the Shapiro–Wilk test of normality to determine whether the data were normally distributed. To determine whether diagnostic groups were matched, the association between diagnostic groups, age and years of education, respectively, was determined by one-way analysis of variance (ANOVA) for the overall sample. The association between gender and diagnostic group for the overall sample was determined by the chi-square (c2) test of association.
Non-parametric Spearman’s rank order correlations were used to explore the associations between total MoCA score, age and years of education for the control group. The association between total MoCA score and gender for the controls was determined by one-way ANOVA. A general linear model (GLM) was also conducted to explore the associations between total MoCA score (dependent variable) and diagnostic group, age, years of education and the interaction between age and years of education (independent variables) for the overall sample. Homoscedasticity was checked prior and there is sufficient evidence to suggest homogeneity of variance, thereby reducing the risk of a large type I error (Caudill, 1988). Arbitrary stratifications for age and years of education, respectively, were tested using independent samples t-tests for the control group. Once significant stratifications were identified, descriptive analyses were conducted to illustrate the effect of age and years of education on total MoCA scores.
Internal consistency reliability analysis was conducted using Cronbach’s (1951) alpha coefficient (a). To determine discriminant validity, predictive models for each diagnosis (vs. the control group), based on total MoCA score, age, years of education, and the interaction between age and years of education, were developed using multinomial logistic regression. Non-significant covariates and interaction terms were removed from the model to avoid over-fitting.
Ethical considerations
Ethical approval was provided by the Human Research Ethics Committee-Medical (HREC-M) of the University of the Witwatersrand (Wits) for the clinical group data (M190819) and control group data (M140872). The present study received ethical approval from Wits HREC-M (M200631). All ethical principles outlined in the Declaration of Helsinki were honoured (World Medical Association, 2013).
Results
The control group was not well matched to the clinical groups in terms of age [F(2) = 12.30, p < 0.001] and years of education [F(2) = 13.65, p < 0.001]. Post-hoc comparisons using the Bonferroni test revealed that the mean age (in years) of the control group (M = 47.5, standard deviation [s.d.] = 9.2) was significantly higher than that of the MP group (M = 40.5, s.d. = 8.9). However, the mean age of the NCD group (M = 43.5, s.d. = 5.3) did not differ significantly from the control group. The mean years of education of the control group (M = 12.6, s.d. = 2.6) were significantly higher than that of the MP (M = 11.1, s.d. = 1.6) and NCD (M = 10.2, s.d. = 2.2) group. Furthermore, the control group was not well matched to the clinical groups in terms of gender; the proportion of males in the control group (n = 41) was significantly higher than in the MP (n = 17) and NCD (n = 2) groups, c2(2, n = 172) = 10.5, p = 0.005. These results indicate that diagnostic groups were not well matched in terms of age, years of education, and gender.
There was a significant negative correlation between total MoCA score and age, rs = –0.28 (Table 2), indicating that as age increases, total MoCA scores decrease. A positive significant correlation was found between total MoCA score and years of education, rs = 0.38, suggesting that as years of education increases, so do total MoCA scores. Despite the adjustment linked to years of education recommended by the test manual, there was a significant negative correlation between age and years of education (rs = –0.41), indicating that younger individuals in the sample have higher levels of education, and that the adjustment is not sufficient to mitigate the impact of education on the total score of this test. The mean difference between males’ and females’ performance on the MoCA was not statistically significant (F(1) = 0.82, p = 0.368). Taken together, these results indicate that the performance on the MoCA is influenced by age and years of education, not gender.
| TABLE 2: Spearman correlations between age, years of education and total Montreal Cognitive Assessment score. |
Descriptive statistics were used to illustrate the effects of age and years of education on total MoCA scores. Prior to this, relevant stratifications needed to be determined. For age, the 47 participants, aged 50 years and younger, obtained significantly higher mean total MoCA scores (M = 23.34, s.d. = 3.74) compared to the 42 participants, aged 51 years and older, who obtained a mean total MoCA score of 21.55 (s.d. = 3.72) points, t(87) = 2.26, p = 0.026. For years of education, the 23 participants, with 11 or less years of education, obtained significantly lower mean total MoCA scores (M = 20.35, s.d. = 4.14) compared to the 66 participants with 12 or more years of education (M = 23.24, s.d. = 3.43), t(87) = –3.30, p = 0.001. These stratifications were subsequently used in the presentation of descriptive data (Table 3). Note that while the performance of the sample aged between 30 and 50 years with between 7 and 11 years of education was used in the analyses, the number of observations in this category was small (n = 3) and therefore not reported.
| TABLE 3: Descriptive statistics for total Montreal Cognitive Assessment scores stratified by age and years of education. |
According to the GLM (Table 4), the interaction between age and years of education was not significant and therefore removed from the model. Only the effect of years of education [F(1) = 7.1, p = 0.009] was significant. Therefore, least squares (LS) means for total MoCA scores were derived from the GLM for the control group and plotted in Figure 1. The adjusted mean total MoCA scores increased by an estimated 0.37 points for every additional year of education. Adjusted mean total MoCA scores ranged between 20.4 points for individuals with the least years of education (7 years) to 24.8 points for individuals with the highest years of education (19 years) at an average age of 47.5 years.
 |
FIGURE 1: The effect of years of education on total Montreal Cognitive Assessment scores. |
|
| TABLE 4: Summary of the general linear model with total Montreal Cognitive Assessment score as the dependent variable. |
Given the unbalanced diagnostic group sizes and demographic heterogeneity of the sample, the raw mean total MoCA scores may not be representative (Cai, 2014). Therefore, LS means for total MoCA scores were derived from the GLM to provide means adjusted for the unequal observations. The adjusted means were identical to the raw means and not statistically significantly different, F(2) = 0.88, p = 0.416. This suggests that the raw total MoCA scores observed are not a product of the unbalanced group sizes. Furthermore, Cronbach’s alpha was 0.64 indicating moderate internal consistency of the MoCA in this sample.
For the MP versus control group (Table 5), the full regression model revealed that total MoCA score was not significant, whereas age (standardised beta [b] = –0.12) and years of education (b = –0.50) were significant predictors of belonging to the MP group. Similarly, for the NCD versus control group, the full regression model showed that age (b = –0.11) and years of education (b = –0.73) were significant predictors, while total MoCA score was not significant. These results indicate that belonging to control, MP, and NCD groups, respectively, could not be predicted by total MoCA scores, signifying the poor discriminant validity of the MoCA in this sample. However, age and years of education were significant independent predictors of diagnostic group belonging, underscoring the importance of considering these demographic variables when administering and interpreting the MoCA. Given these findings, sensitivity, specificity, and cut-off values for the MoCA were not subsequently calculated. These results may be attributable to the finding that total MoCA scores did not differ significantly between the control, MP and NCD groups, even when adjusted for the unbalanced group sizes.
| TABLE 5: Regression analysis of parameters for each clinical group versus controls. |
Discussion
Effect of age, years of education and gender on total Montreal Cognitive Assessment score
In the present study, there was an inverse relationship between age and years of education. This finding may be indicative of the current post-apartheid era in which younger generations of non-English first language speakers have better access to educational opportunities and are obtaining higher levels (i.e. more years) of education (Laher & Cockcroft, 2013; Watts & Shuttleworth-Edwards, 2016). Gender was not significantly associated with performance on the MoCA, consistent with previous research (Malek-Ahmadi et al., 2015; Rossetti et al., 2011; Santangelo et al., 2015).
While there was a significant correlation observed between age and years of education, the statistical significance of the interaction between these variables was not maintained in the GLM. However, the descriptive data support clinical significance. For instance, individuals aged 51–60 years (with less than 12 years of education) scored a total mean of 20.0 points, whereas participants aged 30–50 years old (with 12 or more years of education) scored three points higher with a total mean of 23.4 points on the MoCA. The effect of age is ambiguous in the literature, with some studies finding that total MoCA scores decrease as age increases (Beath et al., 2018; Rossetti et al., 2011), while others report no association (Robbins et al., 2013). Because of age-related cognitive decline, total MoCA scores may decrease with older age (Freitas et al., 2012). However, this trend can be attenuated by educational attainment. According to the cognitive reserve hypothesis, more years of formal education contribute to greater cognitive preservation, which can mitigate the effects of age-related cognitive decline (Stern, 2009).
The contribution of the level of education on MoCA performance in this study was such that the total MoCA scores ranged between 20.4 points for controls with 7 years of education to 24.8 points for subjects with 19 years of education. This emphasises the importance of considering years of education and potentially qualitative variables related to educational history in all psychometric studies conducted in South Africa (Laher & Cockcroft, 2014; Manly, 2005). Furthermore, it is important to highlight that these results are based on the total score of the MoCA that includes an educational adjustment, which demonstrates that the recommended one-point adjustment is not sufficient to mitigate the effects that educational disparities have on this test.
The control group displayed a performance that was well below the originally recommended cut-off of 26, despite being considerably younger. This cut-off was derived from a predominantly homogenous, Caucasian, western and well-educated sample (Nasreddine et al., 2005). The performance of the current control sample was comparable with an ethnically diverse population-based sample in North America, whereby the mean total MoCA scores ranged between 20.5 points for individuals with less than 12 years of education to 24.8 points for individuals with more than 12 years of education (Rossetti et al., 2011). Rossetti et al. (2011) also found that Caucasians had significantly more years of education and obtained significantly higher mean total MoCA scores than Black people, Hispanic people and other groups. Therefore, differences in mean MoCA scores cannot be attributed to ethnicity, as ethnicity is associated with socioeconomic inequalities, which, in South Africa, are linked to access to resources and education (Brickman, Cabo, & Manly, 2006; Laher & Cockcroft, 2013). In addition, the observed differences in MoCA scores may be because of a lack of cultural equivalence of test items, which can be a source of bias (Robbins et al., 2013). However, cultural equivalence is difficult to obtain, define and measure, particularly in culturally diverse contexts such as South Africa. It is therefore often measured via proxy variables, such as language and education (Ng et al., 2018). In this study, we controlled for participants not having English as a first language and having attended public schooling. In addition, Ng et al. (2018) contends that one way to overcome this bias, although not always possible, may be through efforts to validate the cognitive test locally in a well-defined sample, as we have done.
Numerous studies with differing sample characteristics and contextual factors have reported the original cut-off of 26 to be too stringent, increasing the risk of misclassification and false positives (Conti, Bonazzi, Laiacona, Masina, & Coralli, 2015; Pinto et al., 2018; Wong et al., 2015). The most affected by misclassification are less educated individuals; however, even highly educated individuals, with no cognitive impairment, obtain mean MoCA scores within the ‘abnormal’ range (Elkana et al., 2020; Rossetti et al., 2011). Therefore, low MoCA scores cannot be assumed to indicate cognitive impairment. Given that the literature has consistently demonstrated the effects of age and years of education on MoCA performance (Elkana et al., 2020; Wong et al., 2015), these results highlight the need to abandon the suggested universal cut-off point, particularly in heterogenous samples such as the one investigated in this study.
Internal consistency of the Montreal Cognitive Assessment
Demographic characteristics of the sample, such as age and education, can affect the reliability of an instrument (De Souza et al., 2017). In this study, the MoCA showed moderate yet acceptable internal consistency. While some researchers contend that a minimum alpha value of 0.70 is acceptable (De Souza et al., 2017), Kline (1999) notes that values below 0.70 can realistically be expected when dealing with diverse psychological constructs. This is particularly true regarding the MoCA as it represents a multi-construct scale. Alternatively, the moderate internal consistency could be a result of inconsistent responses to items among the sample (De Souza et al., 2017). This may be because of different levels of English proficiency among the respondents, which was not measured (Laher & Cockcroft, 2013).
Discriminant validity of the Montreal Cognitive Assessment
With regard to the discriminant validity of the MoCA, the final regression model revealed that control, MP and NCD groups could not be distinguished based on total MoCA scores, because all three groups obtained similar mean scores. Contrastingly, in another South African study, HIV status was a significant predictor of total MoCA scores because the HIV group obtained significantly lower mean total MoCA scores (M = 18.62, s.d. = 4.39) relative to the controls (M = 21.67, s.d. = 2.00; Robbins et al., 2013). The discrepancies can be attributed to the use of different samples and sampling methods. For instance, in Robbins et al.’s (2013) study, the number of participants in the HIV-positive (n = 39) and HIV-negative (n = 39) groups were balanced, whereas in the present study, diagnostic group sizes were unbalanced. However, the means adjusted for the unequal observations in each diagnostic category were identical to the raw observed means, suggesting that the total mean MoCA scores obtained are not a product of the unbalanced group sizes.
Another possibility is that the utility of the MoCA does not extend beyond its original scope for detecting MCI and dementia to samples with different socio-demographic characteristics and/or HIV populations experiencing comorbid psychiatric and NCDs. Other studies have highlighted the inadequacy of the MoCA in screening for HIV-associated neurocognitive disorders (HAND; Brito-Marques et al., 2019; Hakkers et al., 2018; Kim et al., 2016). For instance, using combined data from North America and South Africa, Joska et al. (2016) found that the MoCA yielded excellent sensitivity (100%), but poor specificity (22%), in screening for HAD. Joska et al. (2016) also stated that ‘a screener used in the USA [United States of America] should also be useful in Africa […]’ (p. 3). While this may be true for well educated and highly acculturated English first-language speakers in Africa, the same may not apply to less westernised non-English first language-speaking populations (Laher & Cockcroft, 2013; Nyamayaro, Chibanda, Robbins, Hakim, & Gouse, 2019).
Limitations
The South African population is diverse. Therefore, the characteristics of the sample may limit generalisability to dissimilar populations (Murad, Katabi, Benkhadra, & Montori, 2018). The relatively small size of the subgroups also limits the accuracy of inferences that can be drawn, especially for the NCD group (n = 13). One of the limitations of using secondary data is that it reduces the researchers control over some aspects of the research, such as sampling and group matching (Jones, 2010). While groups were matched in terms of having English as a second or third language, diagnostic groups were not well matched in terms of age, years of education and gender. However, archival data facilitate preliminary studies such as this which could otherwise not be conducted during times of elevated risk, such as during a pandemic. In addition, while matching may enhance comparability between cases and controls and improve statistical efficiency, variability in the factors of interest is necessary in studies such as this to evaluate the effect that these variables have on total MoCA scores (Bland & Altman, 1994).
Implications and recommendations
The reliability and validity of the MoCA in South African samples have not been extensively examined. This study has demonstrated that the MoCA may not be accurate for detecting cognitive impairment associated with HIV in populations with similar sample characteristics. Therefore, caution is recommended when utilising the MoCA in the diverse South African context. Clinicians in South Africa require reliable, inexpensive, easily administered screening and diagnostics tools. The MoCA may be a suitable candidate; however, further research is needed. Future studies should endeavour to place stricter controls of variables that have an impact on cognitive performance, such as quality of education, language proficiency, acculturation and socioeconomic status (Brickman et al., 2006; Laher & Cockcroft, 2013). This may serve to improve the representation of other demographic variations to improve the generalisability of findings.
Conclusion
The present study found that performance on the MoCA was significantly influenced by age and years of education. The MoCA showed moderate internal consistency, but poor discriminant validity, suggesting that the MoCA may not be useful as a screening or diagnostic tool for cognitive impairment associated with HIV in populations with similar sample characteristics. More extensive evaluation of the psychometric properties of the MoCA in South Africa is recommended.
Acknowledgements
We are grateful to all the participants who volunteered their time to be part of this study.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
E.K. was responsible for the study concept and design, quality control of the data, data analysis and interpretation, and writing of the manuscript. A.F.-C. was responsible for the study concept and design, organisation, acquisition and capturing of data, supervision of the study, and critical revision of the manuscript for important intellectual content. M.S. was responsible for the organisation and acquisition of clinical data and critical revision of the manuscript.
Funding information
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Data availability
The anonymised data set that supports the findings of this study are available on request from the corresponding author, E.K.
Disclaimer
The article is based on the first author’s (E.K.) thesis, submitted to the University of the Witwatersrand for the Degree of Master of Arts in Psychological Research. The control data reported in this article was collected for the PhD thesis of the second author (A.F.-C.) (Ferreira-Correia, 2019). The Neurocognitive Profile of Huntington Disease-Like 2: A Comparison with Huntington Disease and Healthy Controls (PhD), University of the Witwatersrand, Johannesburg, South Africa. The views expressed in the present article are of the authors alone and not an official position of the institutions they are affiliated with.
References
Ary, D., Jacobs, L.C., Razavieh, A., & Ary, D. (2010). Introduction to research in education (8th ed.). Wadsworth: Belmont, CA.
Beath, N., Asmal, L., Van den Heuvel, L., & Seedat, S. (2018). [ABSTRACT] Validation of the Montreal cognitive assessment against the RBANS in a healthy South African cohort. South African Journal of Psychiatry, 24(0), a1304. https://doi.org/10.4102/sajpsychiatry.v24i0.1304
Bland, J.M., & Altman, D.G. (1994). Statistics notes: Matching. British Medical Journal, 309, 1128. https://doi.org/10.1136/bmj.309.6962.1128
Brickman, A.M., Cabo, R., & Manly, J.J. (2006). Ethical issues in cross-cultural neuropsychology. Applied Neuropsychology, 13(2), 91–100. https://doi.org/10.1207/s15324826an1302_4
Brito-Marques, P.R., Cabral-Filho, J.E., Briano, I.O., Milet, G.M.M., Silva, C.E., Rocha-Filho, P.A.S., & Da Cunha-Correia, C. (2019). Comparison between the Mini-Mental State Examination and Montreal Cognitive Assessment as a Cognitive Screening Tool in Patients with human immunodeficiency virus-associated neurocognitive disorders. Revista Da Sociedade Brasileira de Medicina Tropical, 52, e20190473. https://doi.org/10.1590/0037-8682-0473-2019
Cai, W. (2014). Making comparisons fair: How LS-Means unify the analysis of linear models. SAS Institute Inc. Accessed from https://support.sas.com/resources/papers/proceedings14/SAS060-2014.pdf
Caudill, S.B. (1988). Type I errors after preliminary tests for heteroscedasticity. The Statistician, 37(1), 65–68. https://doi.org/10.2307/2348380
Cecato, J.F., Martinelli, J.E., Izbicki, R., Yassuda, M.S., & Aprahamian, I. (2016). A subtest analysis of the Montreal cognitive assessment (MoCA): Which subtests can best discriminate between healthy controls, mild cognitive impairment and Alzheimer’s disease? International Psychogeriatrics, 28(5), 825–832. https://doi.org/10.1017/S1041610215001982
Cockcroft, K., Alloway, T., Copello, E., & Milligan, R. (2015). A cross-cultural comparison between South African and British students on the Wechsler Adult Intelligence Scales Third Edition (WAIS-III). Frontiers in Psychology, 6(297), 1–11. https://doi.org/10.3389/fpsyg.2015.00297
Conti, S., Bonazzi, S., Laiacona, M., Masina, M., & Coralli, M.V. (2015). Montreal Cognitive Assessment (MoCA)-Italian version: Regression based norms and equivalent scores. Neurological Sciences, 36(2), 209–214. https://doi.org/10.1007/s10072-014-1921-3
Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
De Souza, A.C., Costa-Alexandre, N.M., & De Brito-Guirardello, E. (2017). Psychometric properties in instruments evaluation of reliability and validity. Epidemiologia e Serviços de Saúde, 26(3), 649–659. https://doi.org/10.5123/S1679-49742017000300022
Elkana, O., Tal, N., Oren, N., Soffer, S., & Ash, E.L. (2020). Is the cutoff of the MoCA too high? Longitudinal data from highly educated older adults. Journal of Geriatric Psychiatry and Neurology, 33(3), 155–160. https://doi.org/10.1177/0891988719874121
Etikan, I., Musa, S.A., & Alkassim, R.S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11
Ferreira-Correia, A. (2019). The neurocognitive profile of Huntington disease-Like 2: A comparison with Huntington disease and healthy controls. PhD thesis, University of the Witwatersrand.
Ferreira-Correia, A., Anderson, D.G., Cockcroft, K., & Krause, A. (2020). Single case-control design for the study of the neuropsychological deficits and dissociations in Huntington’s disease-like 2. MethodsX, 7, 100782. https://doi.org/10.1016/j.mex.2020.100782
Ferrett, H.L., Carey, P.D., Baufeldt, A.L., Cuzen, N.L., Conradie, S., Dowling, T., … Thomas, K.G.F. (2014). Assessing phonemic fluency in multilingual contexts: Letter selection methodology and demographically stratified norms for three South African Language Groups. International Journal of Testing, 14(2), 143–167. https://doi.org/10.1080/15305058.2013.865623
Freitas, S., Simoes, M.R., Alves, L., & Santana, I. (2012). Montreal Cognitive Assessment: Influence of sociodemographic and health variables. Archives of Clinical Neuropsychology, 27(2), 165–175. https://doi.org/10.1093/arclin/acr116
Gallo, M.A. (2020). Bantu education, and its living educational and socioeconomic legacy in apartheid and post-apartheid South Africa [Senior Theses], Fordham University. Retrieved from https://research.library.fordham.edu/international_senior/43
Hakkers, C.S., Beunders, A.J.M., Ensing, M.H.M., Barth, R.E., Boelema, S., Devillé, W.L.J., … Van Zandvoort, M.J.E. (2018). The Montreal Cognitive Assessment–Basic (MoCA-B) is not a reliable screening tool for cognitive decline in HIV patients receiving combination antiretroviral therapy in rural South Africa. International Journal of Infectious Diseases, 67, 36–40. https://doi.org/10.1016/j.ijid.2017.11.024
Jones, C. (2010). Archival data: Advantages and disadvantages for research in psychology: Archival data. Social and Personality Psychology Compass, 4(11), 1008–1017. https://doi.org/10.1111/j.1751-9004.2010.00317.x
Jonsson, G., Davies, N., Freeman, C., Joska, J., Pahad, S., Thom, R., … Woollett, N. (2013). Management of mental health disorders in HIV-positive patients. Southern African Journal of HIV Medicine, 14(4), 155–165. https://doi.org/10.4102/sajhivmed.v14i4.50
Joska, J.A., Witten, J., Thomas, K.G., Robertson, C., Casson-Crook, M., Roosa, H., … Sacktor, N.C. (2016). A comparison of five brief screening tools for HIV-associated neurocognitive disorders in the USA and South Africa. AIDS and Behavior, 20(8), 1621–1631. https://doi.org/10.1007/s10461-016-1316-y
Kaya, Y., Aki, O.E., Can, U.A., Derle, E., Kibaroğlu, S., & Barak, A. (2014). Validation of Montreal Cognitive Assessment and discriminant power of Montreal Cognitive Assessment subtests in patients with mild cognitive impairment and Alzheimer Dementia in Turkish population. Journal of Geriatric Psychiatry and Neurology, 27(2), 103–109. https://doi.org/10.1177/0891988714522701
Kim, W.J., Ku, N.S., Lee, Y.-J., Ahn, J.Y., Kim, S.B., Ahn, H.-W., … Kim, E. (2016). Utility of the Montreal Cognitive Assessment (MoCA) and its subset in HIV-associated neurocognitive disorder (HAND) screening. Journal of Psychosomatic Research, 80, 53–57. https://doi.org/10.1016/j.jpsychores.2015.11.006
Kisser, J.E., Wendell, C.R., Spencer, R.J., & Waldstein, S.R. (2012). Neuropsychological performance of native versus non-native English speakers. Archives of Clinical Neuropsychology, 27(7), 749–755. https://doi.org/10.1093/arclin/acs082
Kline, P. (1999). The handbook of psychological testing (2nd ed.). Routledge: London.
Laher, S., & Cockcroft, K. (Eds.). (2013). Psychological assessment in South Africa: Research and applications. Wits University Press: Johannesburg.
Laher, S., & Cockcroft, K. (2014). Psychological assessment in post-apartheid South Africa: The way forward. South African Journal ofe Psychology, 44(3), 303–314. https://doi.org/10.1177/0081246314533634
Lu, J., Li, D., Li, F., Zhou, A., Wang, F., Zuo, X., … Jia, J. (2011). Montreal Cognitive Assessment in detecting cognitive impairment in Chinese elderly individuals: A population-based study. Journal of Geriatric Psychiatry and Neurology, 24(4), 184–190. https://doi.org/10.1177/0891988711422528
Malek-Ahmadi, M., Powell, J.J., Belden, C.M., O’Connor, K., Evans, L., Coon, D.W., & Nieri, W. (2015). Age- and education-adjusted normative data for the Montreal Cognitive Assessment (MoCA) in older adults age 70–99. Aging, Neuropsychology, and Cognition, 22(6), 755–761. https://doi.org/10.1080/13825585.2015.1041449
Manly, J.J. (2005). Advantages and disadvantages of separate norms for African Americans. The Clinical Neuropsychologist, 19(2), 270–275. https://doi.org/10.1080/13854040590945346
Meyer, A.-C.L., Boscardin, W.J., Kwasa, J.K., & Price, R.W. (2013). Is it time to rethink how neuropsychological tests are used to diagnose mild forms of HIV-associated neurocognitive disorders? Impact of false-positive rates on prevalence and power. Neuroepidemiology, 41(3–4), 208–216. https://doi.org/10.1159/000354629
Milanini, B., Ciccarelli, N., Fabbiani, M., Baldonero, E., Limiti, S., Gagliardini, R., … Di Giambenedetto, S. (2016). Neuropsychological screening tools in Italian HIV+ patients: A comparison of Montreal Cognitive Assessment (MoCA) and Mini Mental State Examination (MMSE). The Clinical Neuropsychologist, 30(Suppl 1), 1457–1468. https://doi.org/10.1080/13854046.2016.1183048
Murad, M.H., Katabi, A., Benkhadra, R., & Montori, V.M. (2018). External validity, generalisability, applicability and directness: A brief primer. BMJ Evidence-Based Medicine, 23(1), 17–19. https://doi.org/10.1136/ebmed-2017-110800
Nasreddine, Z.S., Phillips, N.A., Bacdirian, V., Charbonneau, S., Whitehead, V., Collin, I., … Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53(4), 695–699. https://doi.org/10.1111/j.1532-5415.2005.53221.x
Ng, K.P., Chiew, H.J., Lim, L., Rosa-Neto, P., Kandiah, N., & Gauthier, S. (2018). The influence of language and culture on cognitive assessment tools in the diagnosis of early cognitive impairment and dementia. Expert Review of Neurotherapeutics, 18(11), 859–869. https://doi.org/10.1080/14737175.2018.1532792
Nyamayaro, P., Chibanda, D., Robbins, R.N., Hakim, J., & Gouse, H. (2019). Assessment of neurocognitive deficits in people living with HIV in Sub Saharan Africa: A systematic review. The Clinical Neuropsychologist, 33(Suppl 1), 1–26. https://doi.org/10.1080/13854046.2019.1606284
Pinto, T.C.C., Machado, L., Bulgacov, T.M., Rodrigues-Júnior, A.L., Costa, M.L.G., Ximenes, R.C.C., & Sougey, E.B. (2018). Influence of age and education on the performance of elderly in the Brazilian version of the Montreal Cognitive Assessment battery. Dementia and Geriatric Cognitive Disorders, 45(5–6), 290–299. https://doi.org/10.1159/000489774
Rademeyer, M., & Joubert, P. (2016). A comparison between the Mini-Mental State Examination and the Montreal Cognitive Assessment Test in schizophrenia. South African Journal of Psychiatry, 22(1), 1–5. https://doi.org/10.4102/sajpsychiatry.v22i1.890
Robbins, R.N., Joska, J.A., Thomas, K.G.F., Stein, D.J., Linda, T., Mellins, C.A., & Remien, R.H. (2013). Exploring the utility of the Montreal Cognitive Assessment to detect HIV-associated neurocognitive disorder: The challenge and need for culturally valid screening tests in South Africa. The Clinical Neuropsychologist, 27(3), 437–454. https://doi.org/10.1080/13854046.2012.759627
Rossetti, H.C., Lacritz, L.H., Cullum, C.M., & Weiner, M.F. (2011). Normative data for the Montreal Cognitive Assessment (MoCA) in a population-based sample. Neurology, 77(13), 1272–1275. https://doi.org/10.1212/WNL.0b013e318230208a
Santangelo, G., Siciliano, M., Pedone, R., Vitale, C., Falco, F., Bisogno, R., … Trojano, L. (2015). Normative data for the Montreal Cognitive Assessment in an Italian population sample. Neurological Sciences, 36(4), 585–591. https://doi.org/10.1007/s10072-014-1995-y
Shuttleworth-Edwards, A.B., Kemp, R.D., Rust, A.L., Muirhead, J.G.L., Hartman, N.P., & Radloff, S.E. (2004). Cross-cultural effects on IQ test performance: A review and preliminary normative indications on WAIS-III test performance. Journal of Clinical and Experimental Neuropsychology, 26(7), 903–920. https://doi.org/10.1080/13803390490510824
Statista. (2018). Distribution of languages spoken by individuals inside and outside of households in South Africa 2018. Retrieved from https://www.statista.com/statistics/1114302/distribution-of-languages-spoken-inside-and-outside-of-households-in-south-africa/
Stern, Y. (2009). Cognitive reserve. Neuropsychologia, 47(10), 2015–2028. https://doi.org/10.1016/j.neuropsychologia.2009.03.004
UNAIDS. (2019). Country factsheets: South Africa. UNAIDS. Retrieved from https://www.unaids.org/en/regionscountries/countries/southafrica
Watts, A.D., & Shuttleworth-Edwards, A.B. (2016). Neuropsychology in South Africa: Confronting the challenges of specialist practice in a culturally diverse developing country. The Clinical Neuropsychologist, 30(8), 1305–1324. https://doi.org/10.1080/13854046.2016.1212098
Wong, A., Law, L.S.N., Liu, W., Wang, Z., Lo, E.S.K., Lau, A., … Mok, V.C.T. (2015). Montreal Cognitive Assessment: One cutoff never fits all. Stroke, 46(12), 3547–3550. https://doi.org/10.1161/STROKEAHA.115.011226
World Medical Association. (2013). World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. Journal of the American Medical Association, 310(20), 2191–2194. https://doi.org/10.1001/jama.2013.281053
Zhang, H., Zhang, X.-N., Zhang, H.-L., Huang, L., Chi, Q.-Q., Zhang, X., & Yun, X.-P. (2016). Differences in cognitive profiles between traumatic brain injury and stroke: A comparison of the Montreal Cognitive Assessment and Mini-Mental State Examination. Chinese Journal of Traumatology, 19(5), 271–274. https://doi.org/10.1016/j.cjtee.2015.03.007
|
Email this article (Login required)
