Questionable relevance of the person-environment approach to career assessment in South Africa
Although the DAT, 16PF and SDS are used widely in South Africa, research pertaining to these tests raises concerns regarding their relevance (and thus the relevance of the person-environment fit approach) in this context (see De Bruin & Taylor, 2013; Gevers, Du Toit, & Harilall, 1995; Van Eeden & De Beer, 2013; Van Eeden et al., 2013). Based on the literature, the relevance of this approach can be considered in terms of the following subthemes: inadequate reliability and validity of tests in the South African context, Western-based theoretical underpinnings, language and socio-economic bias, and inadequate norms for the South African context.
Inadequate reliability and validity of tests in the South African context
The existing evidence for the DAT (Coetzee & Vosloo, 2000; Owen, 2000; Patel, 2004), 16PF (McDonald & Van Eeden, 2014; JvR Academy, n.d; JvR Psychometrics, 2011; Schepers & Hassett, 2006) and SDS (Du Toit & De Bruin, 2002; Gevers et al., 1995) generally indicate that these tests demonstrate good reliability and validity. However, minimal research has explored the psychometric properties of these tests, particularly the DAT and SDS, in South Africa. A search on Google, Google Scholar and the search engines linked to EbscoHost using the search terms of ‘Differential Aptitude Test South Africa’, ‘DAT’, ‘DAT South Africa’, ‘aptitude tests in South Africa’ and variations of the same indicated that there is a paucity of research on all four forms of the DAT and consequently its psychometric properties. Literature has also specifically argued that little research has explored the reliability and validity of Form K of the DAT (Laher & Mokone, 2008). Similarly, there is limited evidence of the psychometric properties of the SDS in South Africa (Du Toit & De Bruin, 2002; Watson, Foxcroft, & Allen, 2007). This indicates that support for the reliability and validity of these tests in this context is lacking.
Furthermore, evidence suggests that the psychometric properties of the DAT, 16PF and SDS may not replicate across different subgroups, rendering the relevance of the tests in South Africa questionable. Firstly, research pertaining to the cross-cultural validity of these tests is concerning. Minimal research has explored the equivalence of cognitive tests (like the DAT) (Meiring, Van De Vijver, Rothmann, & Barrickvan, 2005) and the SDS across groups within the South African context (Allen, 2005; Du Toit & De Bruin, 2002; Watson et al., 2007). This raises concerns regarding the suitability of the DAT and SDS in South Africa’s multicultural context. While there is a more substantial body of evidence regarding the cross-cultural validity of the 16PF in South Africa (see De Bruin & Taylor, 2013; JvR Psychometrics, 2011), some research does not support its cross-cultural validity in this context (Abrahams, 2002; Abrahams & Mauer, 1999b; JvR Academy, n.d.; Schepers & Hassett, 2006; Van Eeden, Taylor, & Du Toit, 1996, as cited in Van Eeden et al., 2013). This suggests that the test may be culturally biased (see JvR Academy, n.d.; Meiring et al., 2005; Prinsloo & Ebersöhn, 2002). The cross-cultural applicability of the 16PF and SDS is particularly concerning as these tests were originally developed in Western contexts (Maree, 2013).
Western-based theoretical underpinnings
The theoretical underpinnings of the 16PF and SDS further suggest that these tests may be culturally biased in South Africa. The 16PF may be culturally biased in South Africa as indigenous manifestations of personality may not be measured in Western-based tests (Van Eeden & Mantsha, 2007). People from non-Western backgrounds may attach different meanings to the constructs being assessed by this test (Van Eeden & Mantsha, 2007). For example, research suggests that the Warmth factor in the 16PF does not manifest within the Tshivenda culture in terms of open expression of feelings, as they are an emotionally reserved culture (Van Eeden & Mantsha, 2007). Similarly, the cultural validity and relevance of Holland’s model and the SDS in South Africa is questionable as the meaning ascribed to the six broad interests of Holland’s model in this context may differ from that in the USA (Du Toit & De Bruin, 2002). For example, the collectivistic value of ‘ubuntu’ (which emphasises helping others over oneself) can alter the meaning ascribed to career interests (Du Toit & De Bruin, 2002). Interests may play less of a role in cultures that value ubuntu, in comparison to cultures where individualism is emphasised (Du Toit & De Bruin, 2002). The limited research that has explored the suitability of the SDS and Holland’s theory in South Africa is equivocal, with some research supporting the cross-cultural validity of Holland’s model in South Africa while other research does not (see Du Toit & De Bruin, 2002; Morgan, De Bruin, & De Bruin, 2015a, 2015b; Watson et al., 2007). Consequently, the relevance of Holland’s theory and the SDS for certain South African population groups is questionable (Du Toit & De Bruin, 2002; Morgan et al., 2015b) and its use can result in unfair treatment of clients (Morgan & De Bruin, 2017).
Language and socio-economic bias
Research further suggests that the DAT, 16PF and SDS may be biased in terms of language. As test-takers who are completing a psychometric test in a second language tend to first mentally translate a test question into their home language and then select an answer, the performance of second language test-takers on the DAT (a timed test) may be negatively affected by this process (Kgosana, 2017). This is supported by South African research which found population mean score differences on the precursor to the DAT-K (Owen, 1991, as cited in Laher & Mokone, 2008) and significant differences in performance on the DAT-K between English and African first language Black South African students (Laher & Mokone, 2008). Research conducted with the latter sample also found the coefficients for the verbal subtests to be lower than those reported in the test manual (Laher & Mokone, 2008). South African students have also criticised the language used in the DAT-K (Bischof & Alexander, 2008). English language proficiency has also been shown to differentiate between learners’ scores on the DAT-S (MacFarlane, 2006). Second language test-takers may thus perform lower than first language test-takers (MacFarlane, 2006); this is a concern since the DAT is available in only two of the 11 official languages in South Africa (English and Afrikaans; Differential Aptitude Tests – Forms R, S, K and L, n.d.). These lower scores would be a function of the language of the test and not their cognitive functioning (Kgosana, 2017), making it difficult to know if test results reflect language problems or actual ability (Laher & Cockcroft, 2013). Similarly, language competency has been said to affect South African test-takers’ responses to the 16PF (JvR Academy, n.d.). Research investigating the understanding of vocabulary used in the 16PF-SA92 (Abrahams & Mauer, 1999a) and 16PF5 (McDonald & Van Eeden, 2014) found that first language English and Afrikaans speakers scored significantly higher than second language speakers. Poor internal consistency reliability coefficients were also found on the 16PF5 among a sample of Tshivenda first language speakers (Van Eeden & Mantsha, 2007). The findings suggest that the test may be biased against second language English or Afrikaans speakers, where language proficiency affects test performance (Abrahams & Mauer, 1999a; McDonald & Van Eeden, 2014). This may also account for the lack of construct equivalence found across language groups (Meiring et al., 2005; Van Eeden & Mantsha, 2007) While translation of the test into different languages may address these issues, attempts to translate the 16PF5 into Zulu and Tshivenda have faced several challenges. These challenges included: the presence of different Zulu dialects across different regions, retaining the original meaning of test items and the absence of equivalent words and expressions in Zulu and Tshivenda (JvR Academy, n.d.; McDonald & Van Eeden, 2014; Van Eeden & Mantsha, 2007). The difficulty in translating these tests so that they are appropriate for the South African context raises further concern regarding the test’s relevance in this context. The SDS may also suffer from language bias, which casts doubt on the relevance of this test in South Africa. Research showing poor fit between South African test-takers’ SDS scores and Holland’s model has been argued to be a function of test-takers experiencing difficulty in fully comprehending the meanings of test items (Du Toit & De Bruin, 2002).
The effect of socio-economic factors on scores obtained on the DAT, 16PF and SDS also raises concern regarding the relevance of these tests in South Africa. In terms of the DAT, the measurement of aptitudes assumes that everyone who has taken the measure has had the same exposure to these aptitudes; however, this may not necessarily be the case (Puchert, Dodd, & Viljoen, 2017; Van Eeeden & De Beer, 2013). Students from disadvantaged educational backgrounds are unlikely to have been exposed to the same knowledge and skills as students from advantaged backgrounds (Kgosana, 2017). Although the DAT-K is suitable for learners from disadvantaged educational backgrounds (Laher & Mokone, 2008), this may negatively affect performance on cognitive tests like the DAT (Puchert et al., 2017). These results may be erroneously interpreted as poor cognitive ability as opposed to lack of skills or experience in comparison to advantaged peers, possibly leading to the unfair use and interpretation of test results (Puchert et al., 2017). Insufficient exposure and disadvantaged educational background have also been reported to affect performance on the 16PF5, where this had led to tests-takers experiencing difficulty with certain items (Van Eeden & Mantsha, 2007). Socio-environmental factors may also partly affect SDS tests scores. The apartheid legacy of social inequality and employment and other labour conditions (which are likely to differ to those found in Western contexts) may shape how test-takers in South Africa perceive the world of work and thus respond to the SDS (Du Toit & De Bruin, 2002; Morgan & De Bruin, 2017; Van Wijk & Fourie, 2017; Watson, McMahon, & Longe, 2011). For example, socio-economic status may be a more influential factor in South Africans’ career decisions than career interests (Van Wijk & Fourie, 2017). These factors may yield response patterns that depict poor fit between the test results and Holland’s model (Du Toit & De Bruin, 2002; Morgan et al., 2015a). These tests thus do not consider the impact that sociocultural factors have on career-related decisions.
Given the possible biases and challenges faced in using the DAT, 16PF and SDS in South Africa, the relevance of these tests in this context is thus questionable. This is particularly because this can lead to test results that provide unreliable and invalid reflections of test-takers, which can in turn lead to inappropriate career guidance and counselling (McDonald & Van Eeden, 2014; Prinsloo & Ebersöhn, 2002; Wallis & Birt, 2003).
Inadequate norms for the South African context
The relevance of the DAT, 16PF and SDS is further questionable given the issues regarding the norms of these tests. The norms for the DAT and SDS can be considered outdated. In terms of the DAT, the most recent norms available for most DAT forms were published in 2000 (Differential Aptitude Tests – Forms R, S, K and L, n.d.). In terms of the SDS, the South African version of the test is outdated (a problem in and of itself as the test does not align with the current world of work), thus suggesting that the norms are outdated as well (Du Toit & De Bruin, 2002; Van Wijk & Fourie, 2017). This is concerning as outdated norms do not align with societal changes that affect test performance and thus can result in inaccurate or non-meaningful interpretations (Kaplan & Saccuzzo, 2017). Furthermore, information available on the norms of the DAT and 16PF suggest that these norms are not representative of the South African population (see Differential Aptitude Test Form K, n.d.; JvR Academy, n.d.; JvR Psychometrics, 2016). The standardisation of these tests also failed to consider factors that influence test scores such as quality of education and language (Kgosana, 2017; Wallis & Birt, 2003). This is problematic as these issues can also lead to erroneous, negative interpretation and use of test scores (Kaplan & Saccuzzo, 2017).
Alternative approaches to career counselling assessment in South Africa
Considering the challenges currently facing the person approach to career counselling assessment in South Africa, alternative approaches to career counselling assessment in this context have been proposed to enable a more relevant assessment practice (Maree, 2013; Watson & Stead, 2002). The following subthemes were identified: Development of emic tests, qualitative assessment approaches and integrated assessment approaches.
Development of emic tests
For instance, researchers and practitioners have engaged in developing and implementing new and contextually relevant assessments that can be utilised in career counselling (Maree, 2013; 2016; McDonald & Van Eeden, 2014). Some examples of such tests are the South African Personality Inventory (Van Eeden & Mantsha, 2007) and local interest instruments like the Maree Career Matrix and South African Career Interest Inventory (Maree & Taylor, 2016; Morgan et al., 2015a, 2015b). These instruments are not without limitations. For example, despite the South African Career Interest Inventory being a few years into development, research on the South African Career Interest Inventory is limited by unrepresentative, homogenous and small samples, and issues of bias and validity require further exploration (Morgan et al., 2015a, 2015b). Research regarding the possible cultural and linguistic biases of these tests is also lacking (Allen, 2005; Maree, 2010; Rabie & Naidoo, 2019). Hence, the relevance of these tests is not well established.
Qualitative assessment approaches
Another alternative to the person-environment fit approach is the adoption of qualitative approaches to career counselling assessment (Albien & Naidoo, 2016; Morgan, 2010). Unlike the quantitative tests that comprise the person-environment fit approach, qualitative approaches allow for the effect of socio-environmental factors on career development in the country to be considered (Albien & Naidoo, 2016; Buthelezi, Alexander, & Seabi, 2009). These approaches additionally provide the client with an opportunity to take on a more active role in the assessment, helping them to gain exposure to skills that they can use in future career-related decisions and thus develop themselves (Morgan, 2010; Watson & McMahon, 2013). In doing so, it removes the notion that assessment practitioners are the all-knowing experts (Maree, 2015), a notion that can have negative implications given the political history of assessment in South Africa (Bischof & Alexander, 2008). There are various qualitative approaches that have been put forward. For instance, the Systems Theory considers the role of intrapersonal, interpersonal and broader socio-environmental elements in clients’ career development, and can make use of appropriate qualitative assessment instruments such as the My System of Career Influences (Albien & Naidoo, 2016; De Bruin & De Bruin, 2013). Similarly, the Career Construction Theory focuses on clients’ life themes and construction of themselves and their careers. An offshoot of this theory is life-design counselling, which focuses on clients’ life and career narratives (Cook & Maree, 2016). Examples of instruments that can be used in these approaches are: the Career Style Interview (which is used to encourage clients to narrate and find meaning in their career stories; De Bruin & De Bruin, 2013) and the narrative component of the Career Interest Profile (a locally developed qualitative instrument that provides narrative data regarding the client’s career interests; Maree, 2017).
It should be noted that while qualitative assessment approaches help to address some of the limitations of the person-environment approach, this does not negate the utility of quantitative psychometric tests (Di Fabio & Maree, 2013; Morgan, 2010). There are other limitations pertaining to the above-mentioned qualitative assessment approaches. For example, although the My System of Career Influences has been successfully used with South African clients, improvements are required particularly in terms of translating the instrument into different official languages (see Albien & Naidoo, 2016; Watson & McMahon, 2013). There is also a lack of locally relevant qualitative assessments that have been developed (Maree & Beck, 2004; Watson & McMahon, 2013). An additional limitation of these qualitative approaches is that they can be time and labour intensive (Maree & Beck, 2004; Watson & McMahon, 2013).
Integrated assessment approaches
An alternative approach to career counselling assessment that has more recently been advocated for is the integration of quantitative and qualitative assessment approaches (Maree, 2015, 2017). An integration of quantitative and qualitative approaches would involve obtaining clients’ objective psychometric test scores and subjective accounts of their career and life stories (Maree, 2015). Information obtained from both approaches is then combined and drawn upon together during the counselling process (Watson & McMahon, 2013). This integration can be facilitated using instruments such as the Career Interest Profile, which comprises both quantitative and qualitative components (Di Fabio & Maree, 2013).
The integration of quantitative and qualitative approaches to assessment provides a comprehensive approach that will best meet the needs of clients (Maree & Morgan, 2012). It allows for assessment results to be triangulated, thus eliciting more reliable and valid results (Maree, 2013; 2015). In this way, the limitations of using older assessment approaches or using quantitative and qualitative approaches in isolation within the South African context may be addressed (Maree, 2010; Maree & Beck, 2004). Some studies have conducted research using an integrated assessment approach (cf. Maree, 2014, 2018, 2019; Maree, Gerryts, Fletcher & Olivier, 2019; McMahon, Watson, & Zietsman, 2018; Naidoo et al., 2019). While these show the promise of an integrated assessment approach to career counselling in the South African context, these studies are limited by the lack of longitudinal research, lack of translated instruments or use of translators with samples who are non-English first language speakers, use of a few specific quantitative and qualitative assessments, and lack of diverse samples.