UKCAT among the pigeons

The short life of the UK clinical aptitude test (UKCAT) has been a troubled one. The test, which some universities use to help select entrants to medical school, has been criticised for its lack of validity, an error in the 2007 marking system, and the fee imposed on candidates. And UKCAT is set to come under more pressure as the concerns are highlighted in 10 motions due to be debated at the BMA Medical Students Conference in Cambridge on 4 and 5 April.

An “appalling” error that led to the scrapping of a quarter of the 2007 test means thousands of candidates trying to get into medical schools this year may have been disadvantaged by being given one score, while the schools to which they applied were given another, says one motion. Another calls into question the reliability of UKCAT system administrator Pearson Vue, after its sister company, Pearson Driving Assessments, was embroiled in a row over the loss of the personal records of three million learner drivers.

Ian Noble, chairman of the BMA students committee, said: “Any test you put in place has to be evidence based and show it selects the most appropriate people for the job. We have serious concerns that UKCAT doesn’t do that. I’ve yet to see that it does.

“We’re also very concerned about the huge under-representation of medical students from economically worse off and inner city backgrounds. I don’t think putting up another barrier to applicants with the introduction of an extra charge is going to help that situation. I fear it might put a lot of potential applicants off. Studying medicine should be about ability and aptitude and not about debt, or whether you’re prepared to pay to apply.”

What is UKCAT?

UKCAT is run by a coalition of 26 university medical and dental schools in partnership with Pearson Vue, a computer based testing business. The test was introduced two years ago to try to ease the difficult process of selecting the strongest candidates from around 20,000 applicants chasing just under 8000 places. Many have straight A exam results, glowing references, and strong personal statements in their university applications.

Aimed at assessing aptitude rather than educational achievement, UKCAT is taken at high street test centres and last year cost between £60 (€78; $120) and £75. The test comprises four parts—verbal reasoning, quantitative reasoning, abstract reasoning, and decision analysis—and candidates get an instant score before applying to medical schools. Last December, however, it emerged that scores from the abstract reasoning section had been ditched after it was discovered that too many candidates had achieved high marks.

Sixth former Abderahman Kamaledeen has been offered a place to study at King’s College London, part of the UKCAT consortium. He went along to the same computer testing centre where he sat the theory part of his driving test to take UKCAT last year.

“Everyone applying to medical school is anxious enough, and when I got the email saying there’d been a problem with the UKCAT marking system, it just heightened my anxiety even more. There’s ruthless competition for places; it’s cut throat.”

Kamaledeen’s sixth form head, David Ryan, said it was a huge disappointment that a recognised examination body could change results after they were published. “I have two students in my year group who are utterly dejected about the downgrading of their UKCAT result” he said.

“The decision brings into question the whole validity of UKCAT as an accredited exam. How can future candidates have faith in the test if errors such as this one are being made and candidates’ results changed? I would hope that measures will be put in place so that future cohorts will avoid the fiasco that we have seen this year. UKCAT needs to restore its reputation.”

Ian Johnson, subdean of Nottingham University and chairman of the UKCAT board and consortium, thinks that leaving out one section of marks last year didn’t have much effect on candidates. “I can’t speak for all of the other medical schools, but in Nottingham we look at UKCAT more formally than many. Losing a section made a difference of one to two marks out of 100. The only candidates affected were borderline ones. But we interviewed 40 or 50 more people this year to ensure nobody was disadvantaged.” Pearson Vue has added new checks in preparation for next year, he said.

Selection lottery

Each school uses its own procedures to select students and some are more transparent than others. Those signed up to UKCAT can use its scores in any way they want. Others use different aptitude tests or none at all, and not all schools interview candidates.

To add to this patchwork of procedures, there is a lack of uniformity about how universities evaluate the combination of exam results, test scores, university application (UCAS) statements, and teachers’ references when deciding whether to offer a candidate either an interview or a place.

The UKCAT consortium argues its test has the potential to widen participation and improve fairness in the system, assisting universities to create a level playing field for applicants from diverse educational and cultural backgrounds. Katie Petty-Saphon, UKCAT director and executive director of the Medical School Council, said council members had signed up because they were anxious to select students who would make the best doctors, rather than those who had the best A level results.

“Generally students from private schools are more likely to benefit if we just use the A level grade approach. We definitely want to find out who will make the best doctors of the future and work towards selecting these people,” she said.

But critics question the notion that an unpiloted test without validation will improve fairness. They say it contravenes guidelines laid out by the Supporting Professionalism in Admissions Programme, an initiative financed by UK higher education funding bodies set up to help develop fair admissions policies. The programme states that good admissions tests should have rigorous validation and reliability testing and be supported by statistical and research evidence.¹

UKCAT’s critics include Kate Fayers, a specialist registrar at the Bournemouth Diabetes and Endocrine Centre, and her consultant colleagues David Kerr and Tristan Richardson. They say that most medical students they have met seemed to have the qualities you would want from a doctor—including empathy, a caring nature, intelligence, and compassion. In view of this, they question why universities are intent on introducing the “pseudoscience” and “psychobabble” included in the UKCAT exam, when the evidence base for this approach is almost non-existent.

Do aptitude tests work?

They point out that UKCAT admits that the test’s predictive validity has not been studied, whereas A levels have been shown to predict outcome at undergraduate and postgraduate level.²

“UKCAT have suggested the introduction of this test would be welcomed but by whom? As the number of As achieved at A level continues to rise, medical schools are finding selection increasingly difficult. The Tomlinson report suggested the simple addition of stars to the top grade (A*, A**, A***) would redress the balance,” they said.

A review of use of aptitude tests and A levels by Chris McManus and colleagues warns: “Schools are introducing tests of intellectual aptitude without evidence of appropriateness, accuracy, or added value, making them open to legal challenge.”² But it also concedes that an argument exists for developing and validating tests of non-cognitive variables in selection, including interpersonal communication skills, motivation, and probity.

Professor Johnson of Nottingham University argues that the test formats were trialled by developers. Furthermore, a database running alongside the test will provide the information needed to establish predictive validity and alter admissions processes to improve the whole selection procedure, he said. UKCAT is not expensive when compared to some other admissions tests, and bursaries can be applied for, he added.

Trisha Greenhalgh, professor of primary health care at University College London, believes UKCAT isn’t necessarily a bad idea but that candidates have to be protected at this early stage. “It may have potential as a selection instrument, but we have to treat it as something which is under development and be careful to protect hardworking, very ambitious, and dedicated young people from the impact of this experimental stage. They are the people I worry about.”

It could be valuable if it helped overcome the middle class bias that enabled students from affluent backgrounds to gain an unfair edge in A levels through tutoring, help with personal statements, and privileged access to opportunities, she said.

Royal Free and University College medical school is one of four that use an alternative entrance exam—the biomedical admissions test (BMAT)—along with Oxford, Cambridge, and Imperial College London. UK candidates paid £27.30 to sit the written test last year.

Research into BMAT by the Cambridge University assessment team concludes it is a better predictor than anything else available of performance at year one. They say it is possible to screen out applicants unlikely to take advantage of the best of higher education and shows people without the best A level results who do have the potential to take that advantage. This information can then be factored into the interview and offer process.

International experience

Medical schools in the United States, New Zealand, and Australia have used admissions examinations for some time. The US medical college admissions test, which lasts four hours and 45 minutes, has been revised four times since it was created in 1946. In its current version, the test is designed to assess understanding of basic science concepts, problem solving ability, verbal reasoning, and writing skills.

But a 2007 meta-analysis of research into the test showed only a small to medium ability to predict performance in medical school and medical board licensing exams.³ The study called for the development of screening and selection criteria with improved validity to supplement the test.

Tyrone Donnon, assistant professor in the department of community health sciences at the University of Calgary, who led the research team, said his medical school uses only two of the four subtests when assessing candidates for admission to medical school, after rigorous statistical analysis showed that the other two were of little value.

He is testing a new interview process at Calgary to try to tap into non-cognitive attributes such as compassion and communications skills. “I think there is always that 2-5% of candidates who are not necessarily selected for the right reasons. We’re trying to create a level playing field allowing us to select people with a humanistic approach rather than just high academic achievement,” he said.

I thank Kate Fayers for help with research.

Competing interests: None declared.

1. Supporting Professionalism in Admissions Programme. Admissions tests. www.spa.ac.uk/admission-tests/index.html.
2. McManus IC, Powis DA, Wakeford R, Ferguson E, James D, Richards P. Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. BMJ 2005;331:555-9.
3. Donnon T, Paolucci E, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med 2007;82:100-6.

       

 Printable version    Download PDF    E-mail this to a friend    Respond to this article