Bridging the gap: improving CASPer test confidence and competency for underrepresented minorities in medicine through interactive peer-assisted learning (2024)

Abstract

Background

The Computer-based Assessment for Sampling Personal Characteristics (CASPer) is a situational judgement test (SJT) adopted by medical schools to assess applicants’ interpersonal skills. CASPer applicants must compose their responses to ethical dilemmas, thereby highlighting the applicant’s rationale for ethical decision-making. Minority applicants usually lack access to a network of individuals who can offer guidance and expertise on ethical decision-making. As such, this study investigated the impact of a CASPer coaching program designed for minority applicants.

Methods

A free online intervention was designed to help minority applicants prepare for the CASPer test. The program consisted of 35 learners and three medical student tutors. Important attributes of the 4-week program included free access to a medical ethics book, feedback provision to in-class and homework student responses, and facilitation of a mock CASPer. Course feedback was collected. Additionally, a pre and post-program survey was administered to assess learners’ competence and confidence surrounding CASPer test-taking.

Results

Our pre and post-program survey showed significant student improvement in familiarity with the test, increased competence, confidence and preparedness, as well as reduced anxiety (p < 0.05).

Conclusions

Through peer-to-peer teaching and access to medical student mentors, our program addresses socioeconomic barriers that several minority applicants face when applying to medical school.

Introduction

Medical students and physicians should Canada’s diverse popilation.¹^-⁴ In examining the demographics of first-year Canadian medical students, Dhalla and colleagues found that underrepresented minorities in medicine (URMM) included students who were Black, Indigenous, from rural backgrounds, and students of low socioeconomic status (SES).⁴

The medical school admissions process provides an opportunity to increase diversity.⁵ The process involves academic (e.g., grade-point average, Medical College Admission Test) and non-academic screening (e.g., personal statements, reference letters).⁶ Academic screening predicts scholastic performance, but inadequately predict interpersonal skills.⁷ Non-academic screening assesses interpersonal characteristics, but are resource-intensive and lack construct validity.⁶^,⁷ URMMs face inequities in the admissions process; they underperform on academic measures,⁷^,⁸ face financial barriers, and lack the social capital to augment their non-academic applications.¹⁰ Situational judgement tests (SJTs) offer an alternative.¹¹ SJTs challenge applicants to showcase their approach to real-world scenarios,¹²^,¹³ mitigate differences among demographic groups,⁷^,¹⁴ and demonstrate high reliability and validity in assessing interpersonal qualities.¹⁵

The Computer-based Assessment for Sampling Personal Characteristics (CASPer) is one SJT used by medical schools.¹⁶ CASPer is a 12-section online test, with video and text-based scenarios, whereby applicants have five minutes to respond to three questions per section.¹⁷ While conventional SJTs provide multiple-choice options to ethical dilemmas,¹⁶ CASPer requires written responses, thus showcasing the applicant’s rationale for ethical decision-making.¹⁴ Juster and colleagues examined academic and non-academic scores of medical school applicants across demographic groups and found that URMMs underperformed across all measures,⁷ with academic scores having the greatest disparity. CASPer reduced but did not eliminate demographic inequities in the application process.⁷^,¹⁸ Stronger CASPer performance may be attributed to a coaching effect, as SJTs may be amenable to coaching.¹⁹^-²¹ However, URMMs with low SES face financial challenges when applying to medical schools,²² including affording preparation programs for admissions prerequisites, which may result in lower scores.

To promote diversity in medicine, the University of Ottawa Black Medical Students’ Association collaborated with University of Toronto’s Community of Support (COS) – an initiative that supports URMMs’ journey to medical school – to design a peer-assisted CASPer coaching program for URMMs in August 2019. A pre and post-program survey was administered to evaluate CASPer test competence and confidence. The purpose of this study was to describe the program’s curriculum and implementation, and its impact by reporting the survey results.

Methods

Population

COS students are URMMs who self-identify as Black, Indigenous, Persons of Colour, of low SES and/or with a disability.²³ Thirty-five URMMs from COS, enrolled in the CASPer coaching program. Students were taught by three URMM medical students who successfully completed the CASPer test and matriculated into CASPer-requiring medical schools.

Summary of innovation

The free four-week online program consisted of sessions once per week for two hours. We designed this program independently of Altus Assessment Incorporation, which creates and administers CASPer.¹⁷ In session one, we discussed test structure, taught ethical principles, and provided guidance on structuring responses. Sessions two and three included timed practice scenarios designed by tutors or obtained from online resources.²⁴^-²⁸ Homework following each session included readings,²⁹ timed self-reflection exercises and bioethical case studies.³⁰ The mock CASPer exam was administered during session four. Key aspects of the curriculum included free access to an ethics book²⁹ and detailed feedback following in-class, homework and mock exam responses.

Outcome measures

Students anonymously completed surveys before the first class (pre-course) and following the final class (post-course). Survey parameters included demographics, prep-course history, class attendance, and questionnaires for self-evaluation and course assessment via 5-point Likert scale. Below are sample questions pertaining to self-evaluation and course assessment, respectively.

Please rate the following on a scale of 1 (strongly disagree) to 5 (strongly agree):

‘I am familiar with the structure of the CASPer test’

‘The learning environment was positive and supportive’

Gostelow et al. validated the self-assessment questionnaire for their peer-assisted SJT coaching program.³³ The COS database provided data on participants’ medical school interview invites. We did not obtain students’ CASPer scores as Altus releases scores directly to medical schools.

Data analysis and ethics

Data were presented as the mean of responses with standard deviation. Paired pre- and post-course survey responses were compared with the Wilcoxon signed-rank test. P-value < 0.05 was considered significant. Analyses were performed on GraphPad Prism 6. Institutional REB was waived for this quality assurance study evaluating a medical education innovation.

Results

Among 35 URMM learners, 26 resided in Ontario, and others resided in Alberta (n = 4), New Brunswick (n = 2), Quebec (n = 1) and internationally (n = 2). Twenty-five students (71.4%) completed the pre-course survey and 27 (77.1%) completed the post-course survey. Nine participants (33.3%) had previously completed CASPer, and three (11.1%) had taken a previous prep course.

The course’s impact on self-perceptions is reported (Figure 1). There was statistically significant improvement across all measures, with improvement in familiarity with the CASPer test structure (2.54±1.22 vs 4.50±0.51, p < 0.0001), perceived competence (2.83±0.96 vs 4.21±0.59, p < 0.0001), confidence (2.21±0.88 vs 3.71±0.69, p = 0.0004), and preparedness (2.08±1.02 vs 3.88±0.68, p = 0.0004). Students’ anxiety levels significantly reduced (2.54±1.10 vs 3.63±0.88, p = 0.0048), with more students reporting, “I do not feel anxious about taking the CASPer test” following course completion.

Bridging the gap: improving CASPer test confidence and competency for underrepresented minorities in medicine through interactive peer-assisted learning (1)

Student demographics and medical school interview invitations are detailed in Table 1. Thirty-four students applied to medical school during the 2020-2021 cycle, with 15 (44.1%) students receiving ≥1 interview invite. Seven students (20.6%) received interview invites to CASPer-requiring medical schools.

Table 1.

Participant demographics and census of resulting medical school interview invites within CASPer Coaching Program (n = 35)

	Demographics	n(%)
Place of residence	ontario	26(74.3)
	Alberta	4(11.4)
	New Brunswick	2(5.7)
	Quebec	1(2.9)
	International	2(5.7)
Perior CASPer exposure	Candidate took CASPer test before (n=27)
	No	18(66.7)
	Yes, Once	3(11.1)
	Yes, twice	3(11.1)
	Yes, 3 times or more	3(11.1)
	Candidate enrolled in aCASPer perp course before (n=27)	3(11.1)
	Outcomes	n(%)
Post-course census of interview invites	Candidate apply to medical school this 2020-2021 cycle	34(97.2)
	Candidate received interviews(s)	15(42.9)
	Candidate received interviews(s) for a medical school requiring CASPer in admission	7(20.6)

Open in a new tab

Narrative feedback

Program strengths included course structure and interactivity. Students thought lectures were “well structured,” “interactive” and “helped put knowledge into practice.” Students appreciated having “easy access to tutors,” receiving feedback, and found the technology beneficial.

Areas of improvement included personalized tutorship, increasing class time and diversifying content. Recommendations included “one-to-one [learning] on occasion” and “more original content” for those who had written the test before.

Discussion

This free CASPer coaching program was offered exclusively online and free. There were significant improvements in students’ competence and confidence by course completion, and 20.6% received interview invites from CASPer-requiring medical schools.

Sharma found that medical school applicants were anxious about the time pressures for completing SJTs and its weighting on their application.³⁴ Students in our course received iterative feedback for timed practice scenarios, which may have reduced anxiety levels. Another near-peer-led teaching program for the objective structured clinical examination (OSCE) taught by recent graduates to medical students similarly reported an increase in students’ confidence and preparedness.³⁵ Such improvements are likely attributed to unique teaching styles, familiarity with the examination’s structure, and encouragement from enthusiastic instructors.³⁵^,³⁶

Peer-assisted learning (PAL) in medical education is associated with significant improvements in OSCE and test performances.³⁷ As an added metric of interest, we analyzed interview invites from CASPer-requiring medical schools. As students’ academic and non-academic screening collectively impact students’ application scores,⁶^,¹⁴^,³⁸^-⁴¹ we believe our coaching program improved the non-academic component of URMM applications.

Upstream interventions in the admissions process are valuable in enhancing medical student diversity.⁵^,⁷^,⁴²^,⁴³ Reiter and colleagues recommend upstream interventions like pipeline programs,⁵ which support high-achieving students from marginalized backgrounds with mentorship and education. We essentially designed a pipeline program using the PAL model, allowing for cost-effective and sustainable programming.³⁷ Leveraging technology and mentorship to alleviate socioeconomic barriers, our CASPer coaching program fulfilled COS’s mission to support URMMs’ journey to medical school. Nonetheless, the diversity among URMMs in our program may have influenced the study results. We did not collect information on student’s unique URMM identities, thus doing so opens an avenue for future directions. Further research is required to contextualize the program’s impact in identifying the URMM populations that benefit most.

Notably, 18 years following Dhalla et al.’s 2002 study, Khan et al.⁴³ reiterates that Black, Indigenous, rural and students of low SES remain underrepresented in Canadian medical schools. As such, we encourage national diversity data collection of medical school applicants and matriculants. This information may better equip Canadian medical schools to implement targeted pipeline programs that foster diversity.

Limitations

Students’ self-perceived improvement in confidence and competence cannot be correlated to CASPer exam performance. Although comparing students’ CASPer scores to the mean scores of other test participants is appealing, limitations exist. Tests are completed on different days, with variable scenarios, are scored by different evaluators and can be taken in French or English. These variables affect the utility of using test scores as a standalone parameter in measuring our program’s efficacy.

Conclusion

Our CASPer coaching program is an upstream effort in medical school admissions to foster diversity within Canadian medical schools. Our program has expanded to accommodate seven tutors with three parallel courses supporting 100+ URMMs. Moreover, following course feedback, we have introduced innovative practice scenarios and embedded small group sessions for personalized feedback. Overall, our near-peer-led CASPer coaching program successfully improved test confidence and competence among URMM participants. Such pipeline programs, targeting marginalized populations are necessary to increase diversity in medicine.

Footnotes

Conflicts of Interest: The authors have no conflicts of interest to declare.

Funding: This study received no funding.

References

1.Boelen C, Heck JE. Defining and measuring the social accountability of medical schools [Internet]. Geneva: World Health Organization; 1995. Available from: https://apps.who.int/iris/handle/10665/59441 [Accessed on April 12, 2020] [Google Scholar]
2.The Association of Faculties of Medicine of Canada. The future of medical education in Canada (FEMC): A collective vision for MD education [Internet]. Ottawa ON: The Association of Faculties of Medicine of Canada; 2010. Available from: http://cou.ca/wp-content/uploads/2010/01/COU-Future-of-Medical-Education-in-Canada-A-Collective-Vision.pdf [Accessed on April 14, 2020] [Google Scholar]
3.Canadian Medical Association. Addressing gender equity and diversity in Canada's medical profession: A review [Internet]. Canadian Medical Association; 2018. Available from: https://www.cma.ca/sites/default/files/pdf/Ethics/report-2018-equity-diversity-medicine-e.pdf [Accessed on April 12, 2020]
4.Dhalla IA, Kwong JC, Streiner DL, Baddour RE, Waddell AE, Johnson IL. Characteristics of first-year students in Canadian medical schools. Can Med Assoc J. 2002;166(8):1029-1035.Available from: http://www.cmaj.ca/content/166/8/1029.abstract [Accessed on April 4, 2020] [PMC free article] [PubMed] [Google Scholar]
5.Reiter HI, Lockyer J, Ziola B, Courneya CA, Eva K. Should efforts in favor of medical student diversity be focused during admissions or farther upstream?. Acad Med. 2012;87(4):443-448. 10.1097/ACM.0b013e318248f7f3 [DOI] [PubMed] [Google Scholar]
6.Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78:313–21. 10.1097/00001888-200303000-00016 [DOI] [PubMed] [Google Scholar]
7.Juster FR, Baum R, Zou C, et al. Addressing the diversity–validity dilemma using situational judgment tests. Acad Med. 2019;94(8):1197-1203. 10.1097/ACM.0000000000002769 [DOI] [PubMed] [Google Scholar]
8.Elks ML, Herbert-Carter J, Smith M, Klement B, Knight BB, Anachebe NF. Shifting the Curve: Fostering academic success in a diverse student body. Acad Med. 2018;93(1):66–70. 10.1097/ACM.0000000000001783 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Dwayer K. Low income students face systemic barriers to college access. The Ithacan [internet]. 2017; 84(27):A3-A5.Available from: https://tnuqq21kt870t8n1egkbrmbr-wpengine.netdna-ssl.com/wp-content/uploads/2017/04/04.27.17.pdf [Accessed on April 12, 2020] [Google Scholar]
10.Bok J. The capacity to aspire to higher education: ‘It's like making them do a play without a script’. Crit Stud Educ. 2010;51(2):163-178. 10.1080/17508481003731042 [DOI] [Google Scholar]
11.Tiffin PA, Paton LW, O'Mara D, MacCann C, Lang JW, Lievens F. Situational judgement tests for selection: Traditional vs construct-driven approaches. Med Educ. 2020;54:105–115. 10.1111/medu.14011 [DOI] [PubMed] [Google Scholar]
12.Patterson F, Cousans F, Edwards H, Rosselli A, Nicholson S, Wright B. The predictive validity of a text-based situational judgment test in undergraduate medical and dental school admissions. Acad Med. 2017;92:1250–1253. 10.1097/ACM.0000000000001630 [DOI] [PubMed] [Google Scholar]
13.Lievens F, Buyse T, Sackett PR. The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. J Appl Psychol. 2005;90:442–452. 10.1037/0021-9010.90.3.442 [DOI] [PubMed] [Google Scholar]
14.Altus Assessments. Research from New York medical college shows how CASPer can impact diversity in admissions [Internet]. New York Medical College School of Medicine; 2016. Available from: https://altusassessments.com/case-study/research-new-york-medical-college-shows-casper-can-positively-impact-diversity-admissions/ [Accessed on April 14, 2020]
15.Dore KL, Reiter HI, Eva KW, et al. Extending the interview to all medical school candidates—computer-based multiple Sample evaluation of noncognitive skills (CMSENS). Acad Med. 2009;84(10 Suppl):S9–S12. 10.1097/ACM.0b013e3181b3705 [DOI] [PubMed] [Google Scholar]
16.Dore KL, Reiter HI, Kreuger S, Norman GR. CASPer, an online pre-interview screen for personal/professional characteristics: Prediction of national licensure scores. Adv Health Sci Educ Theory Pract. 2017;22:327–336. 10.1007/s10459-016-9739-9 [DOI] [PubMed] [Google Scholar]
17.Altus Assessments. CASPer [internet]. 2020. Available from: https://altusassessments.com [Accessed on April 14, 2020]
18.Association of American Medical Colleges. Using MCAT data in 2019 medical student selection [internet]. Washington DC: Association of American Medical Colleges; 2018.Available from: https://www.aamc.org/system/files/c/2/462316-mcatguide.pdf [Accessed on April 12, 2020] [Google Scholar]
19.Cullen MJ, Sackett PR, Lievens F. Threats to the operational use of situational judgment tests in the college admission process. Int J Sel Assess. 2006;14(2):142-155. 10.1111/j.1468-2389.2006.00340.x [DOI] [Google Scholar]
20.Lievens F, Buyse T, Sackett PR, Connelly BS. Coaching effects on situational judgment tests. Int J Sel Assess. 2012;20:272-282. 10.1111/j.1468-2389.2012.00599.x [DOI] [Google Scholar]
21.Stemig MS, Sackett PR, Lievens F. Coaching effects on SJT validities. Int J Sel Assess. 2015;23:174-181. 10.1111/ijsa.12105 [DOI] [Google Scholar]
22.Lam J, Phillips C, Okafor I. Covid19: an urgent opportunity to decrease financial barriers to medical school admissions. CMEJ. 2020. 10.36834/cmej.70219 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.MD Program University of Toronto. Community of Support [Internet]. 2020. Available from: https://applymd.utoronto.ca/community-support [Accessed on April 15, 2020]
24.Altus Assessments.Sample CASPer scenarios [Internet]. 2020. Available from:https://altusassessments.com/casper/how-it-works/ [Accessed on April 14, 2020]
25.BeMo Academic Consulting Inc. Free CASPer SIM practice test by BeMo [Internet]. 2020. Available from: https://bemoacademicconsulting.com/free-sample-casper-sim-practice-test [Accessed on April 14, 2020]
26.The Savvy Pre-Med. 3 practice CASPer questions and best responses [Internet]. 2020. Available from: http://www.savvypremed.com/savvy-pre-med/2017/7/17/3-practice-casper-questions-and-best-responses [Accessed on April 15, 2020]
27.CASPer. Tips on how to prepare for the CASPer test [Internet]. 2020. Available from: https://takecasper.com/test-prep/ [Accessed on April 15, 2020]
28.Astroff. Start Practicing! [internet]. 2020. Available from: https://www.caspertest.com/casper-sample-questions/ [Accessed on April 14, 2020]
29.Heébert PC. Doing right: a practical guide to ethics for medical trainees and physicians. Don Mills: Oxford University Press; 2009. 304 p. [Google Scholar]
30.University of Washington Medicine Department of Bioethics and Humanities. Ethics in medicine [Internet]. 2018. Available from: https://depts.washington.edu/bhdept/ethics-medicine [Accessed on April 12, 2020]
31.Daily.co. Ship 1-click video chat faster and smarter [Internet]. 2020. Available from: https://www.daily.co/ [Accessed on August 3, 2019]
32.Slack. Slack brings the team together, wherever you are [Internet]. 2020. Available from: https://slack.com/intl/en-ca/ [Accessed on August 3, 2019]
33.Gostelow N, Soothill G, Vawda S, Annan D. Near peer-assisted learning to improve confidence for medical students' situational judgment tests. Educ Health. 2017;30:215-222. 10.4103/efh.EfH_21_17 [DOI] [PubMed] [Google Scholar]
34.Sharma N. Medical students’ perceptions of the situational judgment test: a mixed methods study. Br J Hosp Med. 2015;76(4):234-238. 10.12968/hmed.2015.76.4.234 [DOI] [PubMed] [Google Scholar]
35.Rashid MS, Sobowale O, Gore D. A near-peer teaching program designed, developed and delivered exclusively by recent medical graduates for final year medical students sitting the final objective structured clinical examination (OSCE). BMC Med Educ. 2011;11(11):1-7. 10.1186/1472-6920-11-11 [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Lockspeiser TM, O'Sullivan P, Teherani A, Muller J. Understanding the experience of being taught by peers: the value of social and cognitive congruence. Adv Health Sci Educ Theory Pract. 2008;13(3):361-372. 10.1007/s10459-006-9049-8 [DOI] [PubMed] [Google Scholar]
37.Williams B, Reddy P. Does peer-assisted learning improve academic performance? A scoping review. Nurse Educ Today. 2016;42:23-29. 10.1016/j.nedt.2016.03.024 [DOI] [PubMed] [Google Scholar]
38.Lievens F, Patterson F, Corstjens J, Martin S, Nicholson S. Widening access in selection using situational judgement tests: evidence from the UKCAT. Med Ed. 2016;50:624–636. 10.1111/medu.13060 [DOI] [PubMed] [Google Scholar]
39.Kreiter CD. A proposal for evaluating the validity of holistic-based admission processes. Teach Learn Med. 2013;25:103–107. 10.1080/10401334.2012.741548 [DOI] [PubMed] [Google Scholar]
40.Monroe A, Quinn E, Samuelson W, Dunleavy DM, Dowd KW. An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s?. Acad Med. 2013;88:672-681. 10.1097/ACM.0b013e31828bf252 [DOI] [PubMed] [Google Scholar]
41.Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM. Roadmap to diversity: integrating holistic review practices into medical school admission processes [internet]. Washington, DC: Association of American Medical Colleges; 2010.Available from: https://store.aamc.org/downloadable/download/sample/sample_id/195 [Google Scholar]
42.Young ME, Razack S, Hanson MD. Calling for a broader conceptualization of diversity: surface and deep diversity in four Canadian medical schools. Acad Med. 2012;87(11):1501-1510. 10.1097/ACM.0b013e31826daf74 [DOI] [PubMed] [Google Scholar]
43.Khan R, Apramian T, Kang JH, Gustafson J, Sibbald S. Demographic and socioeconomic characteristics of Canadian medical students: a cross-sectional study. BMC Med Ed. 2020;151:1-8. 10.1186/s12909-020-02056-x [DOI] [PMC free article] [PubMed] [Google Scholar]

Bridging the gap: improving CASPer test confidence and competency for underrepresented minorities in medicine through interactive peer-assisted learning (2024)

Abstract

Background

Methods

Results

Conclusions

Introduction

Methods

Population

Summary of innovation

Outcome measures

Data analysis and ethics

Results

Figure 1.

Table 1.

Narrative feedback

Discussion

Limitations

Conclusion

Footnotes

References