How the pass mark is set for the Part 1, Part 2 and Part 3 MRCOG exams
The purpose of standard setting
Standard setting is a process designed to ensure that an examination is fair and equitable, and that the same standard is applied across different examination diets. As such, it is a way of ensuring consistency and fairness when defining the pass mark of a particular examination.
Some examination material is inherently more difficult than others, particularly as it could be based on less commonly seen conditions. For example a question on precocious puberty may be more difficult as most trainees will base their answer largely on theory, whereas a question on the management of pre-eclampsia may be easier as most trainees will have clinical experience, as well as their reading to draw upon.
Standard setting takes into account the complexity of a task so that the passing standard is determined by actual candidate performance, rather than determining in advance a set failure rate for the assessment. Therefore the pass mark and pass rate can fluctuate from exam sitting to exam sitting because there is no fixed level or quota.
Standard Setting Methods used at the RCOG
Part 1 MRCOG
Ebel’s Method: This method accounts for the difficulty of each question as well as its relevance to ST3 practice. Ebel’s focuses on determining the proportion of borderline candidates who would respond correctly to each exam question. The borderline candidate is a hypothetical examination candidate who is of reasonable intelligence, possesses an average amount of clinical knowledge and who has done reasonable preparation: a candidate who should be just at the correct standard to pass. For the Ebel’s method, questions are classified as easy, medium or hard and their relevance is classified as essential, important or acceptable which creates nine separate categories. The standard setting panel then decide what percentage of questions they believe a borderline candidate would answer correctly from each category.
Part 2 MRCOG
Angoff Method: This method involves a cohort of subject matter experts (O&G Consultants in NHS practice) evaluating each question and providing an estimate as to how likely the borderline candidate would know the answer. These estimates are averaged and added to a standard error of measurement to determine the final pass mark. The standard error of measurement is a measure of precision of an assessment. It indicates the amount of variability in a test administered to a group and serves a function of both the standard deviation of observed scores and the reliability of the test.
Linear Equating: Linear equating is a means of ensuring comparable standards across subsequent examination diets. This method compares the current results with those of previous diets and compensates for the relative difficulty of the papers by measuring and comparing candidate performance on a small number of “anchored” (repeated) and un-anchored questions.
Part 3 MRCOG
RCOG Procedure: This method is based on taking the number of examiner judgements in a particular examination, and adding this to an additional factor as well as a standard error of measurement (as described above), to set the standard for assessing a competent candidate.
People involved with the standard setting exercise
Standard setters at the RCOG are all in current NHS practice in obstetrics and gynaecology who have been recruited on the basis of their clinical expertise and their experience in delivering medical education. The panel reviews the questions testing knowledge of O&G practice in the UK, bearing in mind the standard a competent trainee should achieve by the end of their core training. They have undergone specific training in standard setting and all have a special interest in medical education and examinations.
Regulating the standards of RCOG exams
The GMC are responsible for regulating Royal College exams and all associated assessment tools and methods. They agree the examination standards and also approve any changes to exam methods, standard setting and the curriculum. The use of standard setting at the RCOG bears no relationship to the percentage of candidates who will succeed in the exam, or to any other external factors. The RCOG works hard to use standard setting is to improve the fairness and validity of the exam process and to set levels of competence for exam success. The GMC is sent an annual report about College pass rates of individual membership exams.