English has become the dominant language of business, public life and higher education (Benzie, 2010; Casale & Posel, 2011; Coleman, 2006; Nunan, 2003). Therefore, formal acquisition of English language skills has become essential for success in both higher education and business contexts to enhance economic opportunities in a multinational and international economy (Bedenlier & Zawacki-Richter, 2015; Prinsloo & Heugh, 2013). Higher education serves an essential role in enhancing the future career prospects in a competitive social and economic framework, making success integral for many young people (Coleman, 2006; Cross & Carpentier, 2009; Prinsloo & Heugh, 2013). Although higher academic success has become essential for entry into the 21st century economy (Jackson, 2015), academic English language proficiency remains a challenge for the majority of South African students in a linguistically diverse society (Andrade, 2006; Cross & Carpentier, 2009; Murray, 2010; Trenkic & Warmington, 2018).

Academic English proficiency in higher education encompasses formal and functional control of the properties of English language, including vocabulary, grammar and contextual understanding (Bridgeman, McBride, & Monaghan, 2004; Masrai & Milton, 2018; Murray, 2010). Limited English proficiency on entry may lead to academic vulnerability, characterised by unsuccessful adaptation to higher education demands, which could be detrimental to academic literacy, problem-solving techniques, constructive engagement in learning processes (Murray, 2010; Taylor & Von Fintel, 2016) and communications (Benzie, 2010; Murray, 2010; Trenkic & Warmington, 2018; Webb, 2002). Concomitantly, lack of capability in basic interpersonal communication skills (BISC; expression of conversational fluency) alongside cognitive academic language proficiency (CALP; decontextualised language proficiency) may synergistically impact the expression of general English language proficiency in multiple contexts (Bruton, Wisessuwan, & Tubsree, 2018; Cummins, 2000). This disadvantage is displayed where decontextualised language learning experiences in everyday learning and communications, linked to BISC, impact the learning of academic concepts, and thereby result in less than optimal CALP (Abriam-Yago, Yoder, & Kataoka-Yahiro, 1999; Tomasello, 2014). Thus, students lacking English language skills sufficient for the tertiary academic environment are placed at a disadvantage, even if basic literacy skills are sufficient.

Apart from basic literacy, the context of higher education often requires content-specific skills (linked to CALP; Cummins, 2000), which are reliant on technical vocabulary beside general contextual identification and understanding (Dalton-Puffer, 2011; Fenton-Smith, Humphreys, & Walkinshaw, 2018; Millin & Millin, 2018). Global research has implied that basic skills are a necessary component for developing technical/academic language (Birrell, 2006; Coleman, 2006). Consequentially, students lacking English proficiency skills, or exhibiting competency gaps, may be at an academic disadvantage on entering English language institutions.

Internationally, English proficiency tests are frequently conducted pre-admission for selection purposes. Although these tests could be utilised for admitting students in first year, they are often time-consuming, expensive and focused on overall proficiency rather than critical basic skills more relevant to post-admissions phase (Arrigoni & Clark, 2015; Feast, 2002; Goto, Maki, & Kasai, 2010; Murray, 2010). These traditional gate-keeping tests include the International English Language Testing System (IETLS) and the Test of English as a Foreign Language (TOEFL). The viability and financial feasibility of utilising these assessments post-admissions to identify competency gaps is insufficient. Post-admissions, other options, including the Diagnostic English Language Test and Diagnostic English Language Needs Assessment, have been used globally for screening and diagnosis with good predictive and diagnostic validity (Doe, 2014; Read, 2008). Similar to pre-admission tests, the foci include vocabulary, speed-reading, listening and interpretation of texts. In both cases, complex, rather than base skills are inherent to the tests. Thus, other research has indicated that briefer, basic ability tests, including Cloze procedure protocols and vocabulary assessments, are time- and cost-effective whilst retaining sufficient psychometric properties (Goto et al., 2010; Sun & Henrichsen, 2010).

Cloze procedure protocols require the reader to insert missing words or phrases, illustrating semantic and contextual understanding linked to reading comprehension and writing skills (Gellert & Elbro, 2013; Trace, Brown, Janssen, & Kozhevnikova, 2017). Such skills are considered essential in higher education and significantly vulnerable for second-language English speakers, perhaps because of inability to decode new information and translate key words within specific contexts (Escamilla, 2009; Huettig, 2015; Staub, Grant, Astheimer, & Cohen, 2015). Decoding, recognition and translation to English (in the case of non-native speakers) have been closely related to Cloze procedure protocol performance in children and adults (Gellert & Elbro, 2013; Keenan, Betjemann, & Olson, 2008). These findings suggest that background and fundamental learning could play a role in developing essential skills which are transferable to higher education English language requirements. Similarly, vocabulary acquisition has been linked to success in the context of higher education.

Acquired vocabulary has often been used as a proxy for general proficiency, demonstrating predictive power (Masrai & Milton, 2018; Trenkic & Warmington, 2018). Non-technical vocabulary levels have been further linked to academic writing, reading comprehension and general academic performance (Harrington & Roche, 2014; Qian, 2002; Schmitt, Jiang, & Grabe, 2011; Snow, Lawrence, & White, 2009; Trenkic & Warmington, 2018). These findings are supportive of the inclusion of vocabulary components in traditional gate-keeping tests, lending support for the use of these tests as a proxy for proficiency even post-admissions in first-year students. In both cases, the feasibility of reduction in time and cost is a significant benefit.

Although research has demonstrated that both Cloze procedure protocols and contextually based vocabulary tests may be used as proxies to understand English proficiency, these assessments are often conducted under time constraints, potentially confounding content performance with response time (e.g. Goto et al., 2010; Harrington & Roche, 2014; Masrai & Milton, 2018). Administration under time-constrained conditions remains a common practice for a variety of reasons but may result in decreased validity and reliability values (Van der Linden, 2011). Concomitantly, the test may then lack accuracy for its stated purpose, which is problematic for both selections and post-admission competency identification contexts. Therefore, a balance between internal consistency, predictive validity, length of assessment and other administration factors is required to enhance identification of the status of English language skills. The question then arises as to whether a sufficient balance of time-effectiveness, practicality and predictive validity is present when time constraints are implemented.

Researchers have reported improvements in performance on various English language tests with additional time allocations (Bridgeman et al., 2004; Powers & Fowles, 1997), suggesting a focus on performance in complex understandings may be more important for academic outcomes than time-constrained responses (Daly & Stahmann, 1968; Harrington & Roche, 2014; MacIntyre & Gardner, 1994). The removal of time constraints may also mitigate other factors associated with poorer performance, including inadequate test-taking strategies, test anxiety and familiarity with testing contexts (Anderson, 1991; Fairbairn, 2007; Solano-Flores, 2008). Similar findings are present in the context of higher education, for which increased predictive validity, reliability and construct validity of Cloze procedure protocols and vocabulary tests have been reported when time constraints are removed (Hajebi, Taheri, & Allami, 2018; Snow et al., 2009; Trace et al., 2017).

Researchers have hypothesised that changes in performance under different time constraints may be linked to the number of items attempted, changes to item structures or content functions operating differently (Luke & Christianson, 2016; Talento-Miller, Guo, & Han, 2013; Van der Linden, 2011). Other research has suggested that increased time may allow for better translation and internal reconstructions of semantics and syntax, although this may only be true for lengthy fragments in Cloze procedure protocols or when a wide range of possible responses is presented (Hajebi et al., 2018; Staub et al., 2015). Although this research has considered Cloze procedure protocols, vocabulary and other English proficiency tests without time constraints, limited published work (e.g. Goto, Maki, & Kasai, 2010) has considered different predictive validity of short assessments under various time constraints.

The present study assessed the relative influence of time limits on two English language proficiency tests, that is, a Cloze procedure protocol and contextual vocabulary assessment, to understand differences in the predictive validity under each time limit in determining first-test academic outcomes. The importance of this study lies in differentiation between English proficiency itself and the impact of time constraints on the expression of that proficiency in predicting academic outcomes. Thus, the study intends to contribute through further understanding English proficiency testing in terms of the potentially detrimental impact of time limitations on test outcomes. These findings are potentially useful in enhancing mass language post-admission screening to improve skills-targeted interventions which are time-efficient and effective.

The Cloze procedure subtest yielded a maximum score of 20, whilst the Vocabulary in Context subtest score was out of a possible 30. First-test subject marks were expressed as a percentage value out of 100 possible points. Table 1 shows the mean values (M) and standard deviations (SD) of variables.

Table 1 shows similar levels of dispersion across different groups and subjects. Performance on the ELSA tests improved when time constraints were reduced but levels of dispersion remained stable despite differing sample sizes. No substantial differences in academic marks were present between the three time-limit groups.

The findings indicated that performance on both ELSA tests improved relatively to increase in time limitations. The statistics demonstrated that increased time limits resulted in a statistically significant improvement in performance, whilst the SD levels of mean values remained stable, suggesting that a consistent dispersion in scores was retained. Therefore, the findings reflected improvements in test outcome predictive quality when time limits are removed, despite the inherent limitations of comparing groups of differing sizes (Rosenthal & Rosnow, 2008). Nonetheless, similar improvements in English test outcomes were found by Hajebi et al. (2018) and Snow et al. (2009). In this regard, Harrington and Roche (2014) and Van der Linden (2011) also suggested that improvements in performance could be related to the more accurate assessment of constructs in the English language, rather than the ability to perform under time constraints. This disparity could be partially because of long-held notion of the influence of time constraints on the number of item responses and internal reliability of English proficiency tests themselves (Evans & Reilly, 1972).

Similar studies have suggested that implementing time constraints could reduce the reliability and validity of psychometric and language tests for a wide variety of constructs (Lu & Sireci, 2007), resulting in the absence of equivalency across instruments (Cronbach & Warrington, 1951). Additionally, a biased presentation of English language ability is present if response levels below certain thresholds occur, or without readjustment of item functions (e.g. Van der Linden, 2011). The present research findings of improved performance without time constraints cannot necessarily be equated to changes in reliability or validity per se because of the absence of measurement of item response functions, despite studies such as those performed by Harrington and Roche (2014) being focused on similar assessment types. Nonetheless, Talento-Miller et al. (2013) also suggested that increasing the number of items attempted influenced the outcome of English language tests because of the varying difficulties and types of items rather than processing speed. The evidence suggests that inherent, internal test-structure issues under time-constrained conditions are influential, and the present findings concurred that working under time constraints could have negatively affected performance on both ELSA tests for this cohort. Although some other research has explored the inherent reliability issues surrounding time limits on English tests, the reviewed literature has not extensively explored the relative impact of differing time limits on predictive validity in the context of higher education. The regression analyses in the present research provided evidence of a predictive component for the two ELSA tests utilised, which strengthened when time limitations were extended or nulled.

The double time-limit and no time-limit groups’ academic performance was positively and significantly correlated with performance on both ELSA tests, whilst the normal time-limit group demonstrated limited predictive power. Interestingly, predictive performance was similar for both double time-limit and no time-limit groups in most of the cases. This finding suggested that item response thresholds, such as those discussed by Van der Linden (2011) and Talento-Miller et al. (2013), could be important for predictive power as well as for internal consistency and reliability of measurement. Therefore, the present academic first-test performance could have been at least a partial function of English language ability, as measured by the ELSA tests. Several English language performance actions applied to the Cloze procedure protocol were used as one of the ELSA tests. However, in the present research, non-technical vocabulary levels measured in the context were found to be better predictors of academic performance.

Non-technical vocabulary levels have been successfully used as predictors in higher education institutions (HEIs) as well as a proxy for general English proficiency and Cloze procedures (Daller & Wang, 2014; Masrai & Milton, 2018; Qian, 2002; Schmitt et al., 2011; Snow et al., 2009; Trenkic & Warmington, 2018). The present study’s findings suggest that vocabulary levels were more important in accurately predicting academic success than the Cloze procedure test, which required semantic manipulation and decision-making within the context of a passage. However, vocabulary ability could be subsumed into a variety of English functions present in the HEI performance requirements, such as lecture participation and development of text understanding and technical vocabulary. Vocabulary may be linked to other aspects of English language performance related to higher education, including deliberate performance and response selection (Macalister, 2010), improved heuristic learning of phrases and lexical translation (Koehn, Och, & Marcu, 2003), speed of translation and decoding within a finite memory capacity system (Sakurai, 2015), and meta-cognitive focus on syntactical awareness beside reformulation between languages in an attempt at better understanding (Jiménez et al., 2015). Known to be influenced by time constraints, some of these factors directly relate to essential skills measured in vocabulary and Cloze procedure tests, including semantic representations, understanding of words in context, reading speed and quality and the ability to manipulate syntactical arguments.

Reported findings that English proficiency tests encompassing vocabulary, grammar and contextual representation are affected by time limitations (e.g. Bridgeman et al., 2004; Murray, 2010) were confirmed by the present research using two ELSA tests. Such content-specific skill development for understanding may require measurement outside of what could generally be considered as the normal, time-constrained and psychometrically focused framework. Furthermore, development of content/subject-specific technical language could also play a role in academic outcomes, particularly if basic levels have not been fully developed as a foundation (Birrell, 2006). The findings of the present research also suggest that time limitations play an important role in performance and predictive validity, beside choice of test for predictive purposes. Removal of time limitations resulted in more accurate prediction of academic success outcomes, and use of the Vocabulary in Context test resulted in the strongest predictive power. These findings suggested that appropriate English proficiency assessment could hinge more on the determination of specific academic weaknesses within English language whilst reducing the role of time limitation as an essential factor in predicting performance. In spite of the various findings suggesting that time constraints impact a variety of factors concerning English proficiency tests, from a practical perspective, it is unlikely that performing lengthy tests without time limits would be practical in the context of real world. Nonetheless, studies suggest that time constraints could alter the psychometric properties of tests in a variety of ways.

In spite of important findings in the present research, the study carried some limitations which created some uncertainty in the interpretation of the results. Groups of unequal sizes, because of the voluntary nature of participation, may have resulted in misrepresentation of values because of the use of parametric statistics in such a case (Rosenthal & Rosnow, 2008). Similarly, small groups and lack of randomisation could have affected the statistical outcomes. An example of this issue could be the negative correlations seen in for the subject of ‘Travel and Tourism Management’, although alternate explanations such as subject content could also account for this anomaly. Nonetheless, inequality in ranges of scores between different variables still resulted in Pearson’s r and a linear regression model being the most suitable choice, albeit imperfect. In addition, it was not possible to fully standardise the English language pre-entry (Grade 12) performance in this case.Therefore, this criterion was only passively standardised as a minimum level through the use of a specific qualification grouping of students. Pre-entry English ability could have impacted the outcome on either of the English proficiency tests, thus introducing bias in the results or impacting the selection of groups in an attempt by the participants to maximise their performance. Nonetheless, it is believed that present language ability, regardless of prior ability, is the most important factor in interpreting the findings, because the intention is to predict academic performance rather than investigate validity of the assessments in question. Furthermore, the results appear to indicate that time limitations imposed on English proficiency tests are of importance in fully applying the concept of language proficiency to higher education outcomes.

The present research findings demonstrate that performance and predictive power on the modified ELSA versions of Cloze procedure and Vocabulary in Context improves when time limits are increased or removed. The findings imply that factors such as item completion thresholds, reading speed, semantic understanding, and translation for decision-making requirements could contribute to negative changes in performance under time-constrained conditions. Therefore, students may possess some of the English language skills associated with academic performance but are unable to demonstrate these skills within the imposed time constraints. Although these findings are useful, they should be treated with caution as current internal reliability and predictive validity data are not available for full assessment and this pilot study was conducted on smaller, unequal sample groups. Nonetheless, it is apparent that the English proficiency as measured by the ELSA could be inaccurately reflected under time-constrained conditions, limiting the ability of the test to serve as a predictor of academic performance in tertiary education. These findings imply that further investigations are required to develop sufficiently competency gap-targeted English interventions, and the future research should consider larger-scale studies to identify specific components within the tests which contribute to academic success in South African HEIs.