Examining the Argument-Based Validity of the HKDSE Speaking Test by Investigating the Domain Definition, Explanation, and Utilization Inferences

PDF(466 KB)

Asian Journal of English Language Teaching ›› 2024, Vol. 33 ›› Issue (1) : 53-86.

Examining the Argument-Based Validity of the HKDSE Speaking Test by Investigating the Domain Definition, Explanation, and Utilization Inferences

Amy KONG

Author information +

History +

Abstract

Second language (L2) testing research over the last two decades has witnessed a noticeable shift from measuring multiple validities to presenting evidence to the argument-based validation framework, but seldom did they examine the argu- ment-based validity of the Hong Kong Diploma of Secondary Education Exami- nation (HKDSE) English Language speaking test (hereafter referred to as HKDSE speaking test), which is taken in the form of group interaction that considers not only the validation components based on Levelt's (1989) speaking model but also international competence. The current study aims to validate the HKDSE speaking test by presenting evidence to back the warrants drawn in the domain description, explanation, and utilization inferences within the argument- based framework. The study adopted the qualitative approach by analyzing different sources of artifacts, including the constructs, task test samples, marking criteria, and authentic group discussion samples, as well as transcripts of the interview with four prospective task-takers. Results indicate that despite high consistency between most constructs, marking criteria, and theoretical expecta- tions, the non-specifications of the interaction context, participants' roles, and task purposes in the majority of the test prompts make it hard to fully justify both the target domain and explanation inferences. The interviewees' disregard of the representativeness of the test scores in manifesting the language use in real-life contexts has also undermined its validity in terms of utilization inference.

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Amy KONG. Examining the Argument-Based Validity of the HKDSE Speaking Test by Investigating the Domain Definition, Explanation, and Utilization Inferences[J]. Asian Journal of English Language Teaching. 2024, 33(1): 53-86

References

Ambele, E. A., & Boonsuk, Y. (2018). Silence of Thai students as a face-saving politeness strategy in a multicultural university context. Arab World English Journal, 9(4), 221-231.
Berry, V. (2007). Personality differences and oral test performance. Peter Lang.
Bui, G., & Kong, A. (2019). Metacognitive instruction for peer review interaction in L2 writing.Journal of Writing Research, 11(2), 357-392.
Cavalari S. M. S., Freschi A. C., & Pereira Filho, S. A. (2022). Peer corrective feedback: Face-saving strategies in teletandem oral sessions.TheESPecialist, 43(1), 1-19.
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple….Language Testing, 29(1), 19-27.
Chapelle C. A., Enright M. K., & Jamieson J. M. (2008). Building a validity argument for the Test of English as a Foreign Language. Routledge.
Chapelle, C. A., & Lee, H. W. (2021). Conceptions of validity. In G. Fulcher & L. Harding (Eds.), The Routledge handbook of language testing(2nd ed., pp. 17-31). Routledge.
Cheng, L., & Sun, Y. (2015). Interpreting the impact of the Ontario Secondary School Literacy Test on second language students within an argument-based validation framework.Language Assessment Quarterly, 12(1), 50-66.
Chung, Y.-R. (2014). A test of productive English grammatical ability in academic writing: Development and validation [Unpublished doctoral dissertation.] Iowa State University.
De Guerrero, M. C., & Villamil, O. S. (2000). Activating the ZPD: Mutual scaffolding in L2 peer revision.The Modern Language Journal, 84(1), 51-68.
Dursun A.,& Li, Z. (2021). A systematic review of argument-based validation studies in the field of language testing (2000-2018). In C. A. Chapelle & E. Voss (Eds.), Validity argument in language testing: Case studies of validation research (pp. 45-70). Cambridge University Press.
Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Orienta-Konsultit.
Field, J. (2011). Cognitive validity. In L. B. Taylor (Ed.), Examining listening: Research and practice in assessing second language listening (pp. 77-151). Cambridge University Press.
Galaczi, E. D. (2008). Peer-peer interaction in a speaking test: The case of the first certificate in English examination.Language Assessment Quarterly, 5(2), 89-119.
Galaczi,E.,&Taylor, L. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions.Language Assessment Quarterly, 15(3), 219-236.
Global Education and Talent Solutions (ETS) (2014). TOEFL speaking rubrics. ETS. www.ets.org/s/toefl/pdf/toefl_speaking_rubrics.pdf
Grabowski, K. (2013). Investigating the construct validity of a role-play test designed to measure grammatical and pragmatic knowledge at multiple proficiency levels. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 149-171). Palgrave Macmillan.
Hansen, J. H., & Liu, J. (2005). Guiding principles for effective peer response. ELT Journal, 59(1), 31-38.
Hong Kong Examinations and Assessment Authority (HKEAA) (2025). HKDSE English language assessment framework. HKEAA. https://www.hkeaa.edu. hk/DocLibrary/HKDSE/Subject_Information/eng_lang/2025hkdse-e-elang.pdf
HKEAA (2021a). Category A: Senior secondary subjects—standards-referenced reporting.HKEAA.https://www.hkeaa.edu.hk/en/hkdse/assessment/the_reporting_system/
HKEAA (2021b). Category A—HKDSE core subjects: English language—age - samples of candidates' performance—2019 exam—Paper 4. HKEAA. https://www.hkeaa.edu.hk/en/HKDSE/assessment/subject_information/category_a_subjects/eng_lang/sp/2019_Paper4.html
HKEAA (2019a). English language paper 4 guidelines for candidates. HKEAA.https://www.hkeaa.edu.hk/DocLibrary/HKDSE/Subject_Information/eng_lang/EngDSESpeakingGuidelines2019.pdf
HKEAA (2019b). HKEAA publications collection.HKEAA. https://mmis.hkpl.gov.hk/hkeaa-collection
HKEAA (2018). HKEAA publications collection. HKEAA. https://mmis.hkpl.gov.hk/hkeaa-collection
HKEAA (n.d.). HKDSE English language paper 4 speaking—Assessment guidelines. HKEAA. https://www.hkeaa.edu.hk/DocLibrary/HKDSE/Subject_Information/eng_lang/EngPaper4-Guidelines.pdf
Hong Kong Education Bureau (EDB) (2021). CDC-HKEAA English language curriculum and assessment guide (secondary 4-6) 2021. EDB. https://www.edb.gov.hk/attachment/en/curriculum-development/kla/engedu/Curriculum%20Document/English%20Language%20Curriculum%20and%20Assessment% 0Guide%20(Secondary%204%20-%206)%20(2021).pdf
Hwang A., Ang S., & Francesco A. M. (2002). The silent Chinese: The influence of face and kiasuism on student feedback-seeking behaviors.Journal of Management Education, 26(1), 70-98.
Hymes, D. (1972). On communicative competence. In J. Pride & J. Holmes (Eds.),Sociolinguistics: Selected readings (pp. 269-293). Penguin.
International English Language Testing System (IELTS) (2017). Speaking band descriptors (public version). IELTS. https://takeielts.britishcouncil.org/sites/default/files/ielts_speaking_band_descriptors.pdf
Jun, H. S. (2014). A validity argument for the use of scores from a web-searchpermitted and web-source-based integrated writing test. [Unpublished doctoral dissertation,] Iowa State University.
Kane, M. T. (2004). Certification testing as an illustration of argument-based validation.Measurement, 2(3), 135-170.
Kong, A. (2019). A dual case study of perceptions of strategy-based training and peer review stances among Hong Kong secondary students: An activity theory perspective. In B. Reynolds & M. Teng (Eds.), Innovative approaches in teaching English writing to Chinese speakers (pp. 43-66). De Gruyter Mouton. Kong, A., & Bui, G. (2019). Reader stances and writer responses in L2 peer review: A study of L2 writing literacy among Hong Kong secondary school
students.Asian EFL Journal, 21(3), 140-186.
Kong, A., & Teng, M. F. (2023). The operating mechanisms of self-efficacy and peer feedback: An exploration of L2 young writers.Applied Linguistics Review, 14(2), 297-328.
Lantolf, J. (2000). Sociocultural theory and second language learning. Oxford University Press.
Lantolf J.,& Thorne, S. L. (2006). Sociocultural theory and the genesis of second language development. Oxford University Press.
Levelt W. J.M. (1989). Speaking: From intention to articulation MIT Press From intention to articulation. MIT Press.
Liu J.,& Hansen, J. G. (2002). Peer response in second language writing classroom. University of Michigan Press.
Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement(3rd ed., pp. 13-103). Macmillan Publishers.
O'Sullivan, B. (2008). Modelling performance in tests of spoken language. Peter Lang.
Park, M. (2021). Domain definition inference for a virtual Interactive aviation English test (VIAET) for military air traffic controllers. In C. A. Chapelle & E. Voss (Eds.), Validity argument in language testing: Case studies of validation research (pp. 73-95). Cambridge University Press.
Pearson Education (2012). PTE Academic score guide. Pearson Education. https://www.pearsonpte.com/ctf-assets/yqwtwibiobs4/5Sz9Ur4qbus8AEOQd etkAj/69f6c1f2e2870980740b10a2ea9b467f/pte-academic-test-taker-score-guide-nov-2024-v4.pdf
Roever, C. (2005). Testing ESL pragmatics. Peter Lang.
Roever, C., & Kasper, G. (2018). Speaking in turns and sequences: Interactional competence as a target construct in testing speaking.Language Testing, 35(3), 331-355.
Santos, V. (2017). A computer-adaptive test of productive and contextualized academic vocabulary breadth in English (CAT-PAV): Development and validation. [Unpublished doctoral dissertation.] Iowa State University.
Voss, E. (2021). The explanation inference for a test of academic English collocational ability. In C. A. Chapelle & E. Voss (Eds.), Validity argument in language testing: Case studies of validation research (pp. 176-203). Cambridge University Press.
Vygotsky L. S.(1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Young, R. (2008). Language and interaction. Routledge.