Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort
- “Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort” by P Tiffin and colleagues (BMJ 2012;344:e1805, doi:10.1136/bmj.e1805)
- Objective—To determine whether the use of the UK clinical aptitude test (UKCAT) in the medical schools admissions process reduces the relative disadvantage encountered by certain sociodemographic groups.
- Design—Prospective cohort study.
- Setting—Applicants to 22 UK medical schools in 2009 that were members of the consortium of institutions using UKCAT as a component of their admissions process.
- Participants—8459 applicants (24 844 applications) to UKCAT consortium member medical schools where data were available on advanced qualifications and socioeconomic background.
- Main outcome measures—The probability of an application resulting in an offer of a place on a medicine course according to seven educational and sociodemographic variables depending on how the UKCAT was used by the medical school (in borderline cases, as a factor in admissions, or as a threshold).
- Results—On univariate analysis all educational and sociodemographic variables were significantly associated with the relative odds of an application being successful. The multilevel multiple logistic regression models, however, varied between medical schools according to the way that UKCAT was used. For example, a candidate from a non-professional background was much less likely to receive a conditional offer of a place compared with an applicant from a higher social class when applying to an institution using the test only in borderline cases (odds ratio 0.51, 95% confidence interval 0.45 to 0.60). No such effect was observed for such candidates applying to medical schools using the threshold approach (1.27, 0.84 to 1.91). These differences were generally reflected in the interactions observed when the analysis was repeated, pooling the data. Notably, candidates from several under-represented groups applying to medical schools that used a threshold approach to the UKCAT were less disadvantaged than those applying to the other institutions in the consortium. These effects were partially reflected in significant differences in the absolute proportion of such candidates finally taking up places in the different types of medical school; stronger use of the test score (as a factor or threshold) was associated with a significantly increased odds of entrants being male (1.74, 1.25 to 2.41) and from a low socioeconomic background (3.57, 1.03 to 12.39). A non-significant trend was seen towards entrants being from a state (non-grammar) school (1.60, 0.97 to 2.62) where a stronger use of the test was employed. Use of the test only in borderline cases was associated with increased odds of entrants having relatively low academic attainment (5.19, 2.02 to 13.33) and English as a second language (2.15, 1.03 to 4.48).
- Conclusions—Using UKCAT could lead to more equitable provision of offers to those applying to medical school from under-represented sociodemographic groups. This might translate into higher numbers of some, but not all, relatively disadvantaged students entering the UK medical profession.
Despite initiatives to widen access to medical training, most medical students come from relatively advantaged backgrounds. No consensus exists as to how to best select medical students from the large number of applicants to medical school every year. Academic results could well be relied on too much, although they have been shown to predict performance at medical schools quite closely. In the United Kingdom, the personal statement on the Universities and Colleges Admissions Service (UCAS) form is widely used as a discriminator, and face to face interviews are thought to be important, but both of these methods have questionable validity.
One innovation is the introduction of the UK clinical aptitude test (UKCAT), which is now used by 25 of the 31 medical schools in the UK. An aptitude test is a selection test (such as an IQ test) in which the candidate’s innate abilities and potential for achievement (and not his or her knowledge) are ascertained. These tests have been used in various forms by certain medical schools for some time. As with other methods of selection, evidence for their validity in selecting students most suitable to become doctors is inconclusive.
One of the reasons for developing UKCAT was to reduce discrimination against applicants from less advantaged backgrounds. At present, however, little evidence exists on how to best use UKCAT in the admissions process, and medical schools vary in how the new test is incorporated and its weighting in the final decision.
What is the research question?
To determine if the use of UKCAT in medical school admission processes reduces the relative disadvantage encountered by certain “under-represented” sociodemographic groups.
What did the authors hope to find?
The authors identified seven educational and sociodemographic variables that were indicative of students who might be less likely to obtain a place in medical school. These include male sex and coming from lower socioeconomic classes. They tried to identify if medical schools that assigned more weight to the UKCAT scores in their admission process were more likely than expected to offer places to, and admit students from, a wider sociodemographic background.
What did the authors do?
The design is a prospective cohort study. A cohort of students applying to medical schools with sociodemographic factors suggesting disadvantage were compared with those students without such factors. Dependent measures were the probability of offers of places to students and entrance to medical school. Independent variables were factors including educational attainment, sex, ethnicity, and UKCAT scores.
Cohort studies are observational and do not involve randomisation of an intervention and are not as robust as a randomised controlled trial. Cohort studies, however, are considered to produce relatively strong evidence if well executed.
In this situation it is impossible to do a randomised controlled trial, so a cohort design is appropriate. The hallmarks of a good cohort study are making sure that as few data as possible are missing, including follow-up of those participants lost to routine data gathering; consideration of the possibilities of confounding and bias; and appropriate analysis (including imputation of missing data where they are irretrievable).
The advantages of a cohort design are that the study is prospective—that is, the cohorts are identified before the outcomes are measured, thus reducing the possibility of bias. Also, in this type of database analysis, large numbers of people can be involved, which increases the precision of the study. In this study, there were more than 24 000 applications with data available, and the number of missing data was comparatively small except in two variables (educational attainment and socioeconomic class), where the authors made imputations of the missing data and carried out specific sensitivity analyses to include these imputations.
It is difficult to conceive of bias creeping into the collection of the exposure or outcome variables because these were routinely collected data. Unrecognised confounding could be a reality, however, because admission processes to medical school are complex, and subtle influences might operate at individual institutions and these cannot have been allowed for.
The analysis is complex, but is necessitated by the richness of the data.
Medical schools vary in use of UKCAT in their admission process
Three categories are identified. Six medical schools use the UKCAT score only for assessing “borderline” cases—called a “weak” use of the test in this paper. In this situation, the test influences decisions on candidates who are doubtful only after all other selection procedures are finished. Nine schools use the score as a factor in deciding whom to interview—called a “moderate” use of the test—but the ultimate decision still depends on interview. Seven medical schools use the score as a threshold and do not consider applicants below the threshold score (called a relatively “strong” use of the test). The threshold can be set at the 25th centile, or the median, of the scores of the previous year’s candidates, for example. The total score, as opposed to any of the subsections, is usually used. The authors have allowed for this variation by carrying out analyses separately for the three different categories of school, and looking for interactions.
Choice of medical school on the UCAS application form
Each candidate can make four choices, and might have up to four offers of a place. The authors have classified each outcome (offer or rejection) as an individual “application event,” and analysed each separately.
Type of offer and acceptance of offer
Offers of a place might be conditional on academic performance, in this case A level results, or might be unconditional. Not all offers are taken up, so that the authors have analysed separately applicant offers and entrants to medical school.
An interaction occurs between two exposure variables if the effect on the outcome of one variable is modified by the other. Investigating possible interactions can produce important insights into the mechanisms underlying associations, and such interactions should be fully reported, as they have been in this paper—for example, the interaction between type of use of UKCAT score and sex of applicant.
Statistical tools such as logistic regression, z scores, and use of intraclass correlation coefficients were used in the analysis of the results. They will not be explored in this article, and further information can be found in the original paper.
What did the study find?
The results suggest that important differences exist—at the level of an offer of a place and at the level of taking up a place at medical school—between candidates with different sociodemographic characteristics, and the way that the medical school uses the UKCAT results. Use of the UKCAT results as a threshold score is the most effective of the three uses in reducing the disadvantage of certain sociodemographic groups. For example, schools using the UKCAT as a threshold score were more likely to admit students who were men (odds ratio 1.74, 95% confidence interval 1.25-2.41), from state or non-grammar schools, (1.6, 0.97-2.62), and from low social economic backgrounds, (3.38, 0.94-12.12), than medical schools using UKCAT in borderline cases (see tables). It should be noted that some of these odds ratios are not statistically significant (because the 95% confidence intervals include 1).
|Predictors||Odds ratio (95% CI)||P value||ICC*|
|UKCAT in borderline cases v as factor||0.65 (0.48 to 0.89)||0.006||0.002|
|UKCAT as threshold v in borderline cases||1.74 (1.25 to 2.41)||0.001|
|UKCAT as threshold v as factor||1.13 (0.85 to 1.46)||0.4|
|Standardised UKCAT total score (SDs)||1.30 (1.15 to 1.46)||<0.001|
|Non-white ethnicity||1.26 (1.02 to 1.55)||0.03|
|Academic attainment (standardised tariff score in SDs)||0.88 (0.77 to 1.00)||0.05|
|English as second language||0.87 (0.65 to 1.18)||0.4|
|Socioeconomic class 4 or 5†||1.06 (0.67 to 1.65)||0.8|
|Age >21 years||0.96 (0.61 to 1.52)||0.9|
|Attendance at state (non-grammar) school||1.00 (0.85 to 1.17)||0.9|
|School type-UKCAT as factor (v in borderline cases) interaction||2.27 (1.51 to 3.41)||<0.001|
|Predictor||Odds ratio (95% CI)||P value||ICC*|
|State or non-grammar school:|
|UKCAT in borderline cases v as factor||0.82 (0.51 to 1.30)||0.5||0.04|
|UKCAT as threshold v in borderline cases||1.60 (0.97 to 2.62)||0.06|
|UKCAT as threshold v as factor||1.31 (0.84 to 2.03)||0.2|
|Age >21 years||3.72 (2.08 to 6.66)||<0.001|
|Male sex||1.67 (1.20 to 2.32)||0.002|
|Socioeconomic class 4 or 5†||1.87 (1.15 to 3.02)||0.01|
|Non-white ethnicity||0.73 (0.57 to 0.94)||0.01|
|Academic attainment (standardised tariff score in SDs)||0.84 (0.73 to 0.98)||0.02|
|English as second language||1.39 (1.03 to 1.89)||0.03|
|Standardised UKCAT total score (SDs)||0.94 (0.82 to 1.07)||0.3|
|UKCAT as factor (v borderline)-male sex interaction||0.44 (0.29 to 0.67)||<0.001|
|UKCAT as threshold (v borderline)-male sex interaction||0.59 (0.39 to 0.90)||0.01|
|UKCAT score-non-white ethnicity interaction||0.70 (0.54 to 0.92)||0.01|
|Socioeconomic class 4 or 5†:|
|UKCAT in borderline cases v as factor||0.28 (0.08 to 0.97)||0.05||<0.001|
|UKCAT as threshold v in borderline cases||3.38 (0.94 to 12.12)||0.06|
|UKCAT as threshold v as factor||0.95 (0.45 to 2.01)||0.9|
|Standardised UKCAT total score (SDs)||0.84 (0.61 to 1.16)||0.3|
|Non-white ethnicity||2.85 (1.33 to 6.13)||0.007|
|Attendance at state (non-grammar) school||1.31 (0.77 to 2.23)||0.3|
|Age >21 years||1.59 (0.55 to 4.58)||0.4|
|English as second language||0.61 (0.20 to 1.81)||0.4|
|Male sex||1.01 (0.64 to 1.59)||0.9|
|Academic attainment (standardised tariff score in SDs)||0.98 (0.72 to 1.34)||0.9|
|School type-English as second language interaction||3.61 (1.05 to 12.41)||0.04|
|UKCAT in borderline cases (v as factor) use-non-white ethnicity interaction||4.66 (1.06 to 20.46)||0.3|
Strengths and limitations
Strengths of the study include the relatively high numbers of participants and the small number of missing data except for the variables social economic background and school qualifications. Relatively few applicants to medical schools came from a lower socioeconomic background (about 5.5%), and this reduces the power of the study to look at the differences in the outcome measures between medical schools for these factors.
It is difficult in a study of this type to control for subtle influences on offers and the take up of places, which vary between schools—for example, differences between interpretation of UCAS personal statements. Whether this confounds the results is up for debate because such a factor would have to be systematically linked to the outcome (that is, the offer and/or acceptance of a place) and to the exposure (that is, method of use of UKCAT) to have an influence. Finally, many statistical tests have been done in this paper and some positive results (that is, statistically significant odds ratios) are to be expected through the play of chance—out of 100 tests done, five will have statistical significance, randomly, at the level of a 95% confidence interval.
What does this mean?
Placing increased emphasis on scores from UKCAT might result in a better chance of “disadvantaged” students getting into medical schools. This study only scratches the surface of a complex subject, however. Questions remain unanswered. Which part of the UKCAT is important (there are four subtest scores available)? What is the best threshold level to set, and does use of the UKCAT result in a “better” doctor at finals and after?
EDITORIAL, p 8; LIFE, p 14
1Highfield Campus, Southampton University, Southampton SO17 1BJ, UK
Correspondence to: email@example.com
Competing interests: None declared.
Provenance and peer review: Commissioned; not externally peer reviewed.
- Panel for Fair Access to the Professions. The final report of the Panel for Fair Access to the Professions. Department for Business Innovation and Skills, 2009.
- Schwartz S. Fair admissions to higher education: recommendations for good practice. Department for Education and Skills, 2004.
- Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teacher 2006;28:335-40.
- McManus I, Powis D, Wakeford R, Ferguson E, James D, Richards P. Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. BMJ 2005;331:555-9.
Cite this as: BMJ 2012;20:e3365