Kristina: Why spend so long on your general practice questions?
Stephen: Because it is the biggest single area of the MLA content map. "General practice and primary healthcare" lists 77 presentations, from abdominal distension to wheeze. It also turns up on both AKT papers, so you can’t leave it to chance.
Kristina: Isn't most of that covered by other questions in the bank already?
Stephen: The topics overlap, but the questions are different. In a GP consultation you are deciding whether to refer, how fast, what to prescribe, and what to tell the patient to come back for. A cough question in a respiratory bank and a cough question in a primary care bank should test different decisions. That’s what you’ll find in the AKT, so that was something we were determined to reproduce in every one of the 77 sets.
Kristina: Where does a question actually start?
Stephen: With the decision we want to test, not with the disease. "Does this woman with a breast lump need a suspected cancer referral?" is a decision. "Breast cancer" is just a topic. Once we select the decision, we create the patient who would make a GP face that decision.
Kristina: What goes into the stem?
Stephen: Every stem follows the same pattern: Age and sex first: "A 58-year-old man". The history a GP would actually hear, in plain language. Observations in a fixed order: temperature, pulse, blood pressure, respiratory rate, oxygen saturation. Any results with the reference range in brackets, so you are never guessing what "normal" means.
Kristina: And the question itself?
Stephen: One short "What...?" lead-in, with one question mark. It has to pass the cover test: cover the options and you should be able to answer it. If you can't, the question is badly written, not hard.
Kristina: How do you choose the wrong answers?
Stephen: The four distractors must be things a reasonable student might actually pick. Usually that means the right action at the wrong time, the right drug for the wrong patient, or a plausible test that won't change management. They are also all the same type: five diagnoses, or five drugs, never a mix.
Kristina: Why are the options always in alphabetical order?
Stephen: That is how the AKT presents them. It also stops us, without meaning to, putting the right answer in the same place every time. Across the bank the correct answer lands fairly evenly across A to E.
Kristina: What's in the explanation?
Stephen: Why the right answer is right, why each distractor is wrong, and the guideline it comes from. Every question has a full web link to its source, usually NICE guidance or a NICE CKS topic. You should be able to check our answers and reasoning.
Kristina: How do you make sure nothing gets missed?
Stephen: The MLA content map is the checklist. We took all 77 primary care presentations in their curriculum order and wrote one set of 20 for each. Nothing was merged or skipped, so fit notes, driving advice and struggling to cope at home get the same 20 questions as chest pain.
Kristina: Twenty questions on one presentation sounds like a lot. How do you stop them repeating?
Stephen: Every set mixes three kinds of question: what is the diagnosis, what is the best investigation, and what is the best management. Within that I spread the conditions. A breathlessness set should take you through asthma, COPD, heart failure, anaemia, pulmonary embolism and anxiety, not ten versions of asthma.
Kristina: Some presentations overlap, though. Dizziness and vertigo, for example.
Stephen: Exactly, and we planned for that. The dizziness set is about feeling faint: postural hypotension, syncope, drugs that drop the blood pressure. The vertigo set is about the spinning kind: BPPV and the Epley manoeuvre, vestibular neuritis, Meniere's disease, and the posterior circulation stroke you must not miss.
Kristina: Do some overlap with your other topics?
Stephen: We kept a running log of every theme already used, in this bank and in the specialty banks. Before writing a set we checked it. If wheeze had already tested life-threatening asthma elsewhere, the primary care wheeze set covered preschool viral wheeze, an inhaled foreign body and the NICE objective-testing pathway instead.
Kristina: How do you catch two questions that are basically the same?
Stephen: At the end we compared every stem in the bank against every other stem by computer, looking for heavy overlap in wording. Where we found true duplication, we replaced the questions. That’s one of the reasons this has taken us so long!
Kristina: How do you make the questions feel like the real exam?
Stephen: By copying its format exactly: a clinical vignette, one lead-in, five options, one best answer. The AKT tests applied knowledge, so almost every question puts you in front of a patient and asks you to act. Very few ask you to recite a fact.
Kristina: How do you decide how hard a question is?
Stephen: Each question is graded against the average UK final-year student. Basic means most final-year students should get it right. There are about 5 of these questions in each set of 20. Medium means a well-prepared student gets it right, and others are split. There are about 10 of these. Advanced needs detailed guideline knowledge or careful reasoning, and there about 5 of these.
Kristina: Some of my fellow students have little tricks, like guessing the answer because it's the longest option. Did you deal with that?
Stephen: Yes, because that is a real flaw in a lot of practice questions. Writers add qualifiers to the right answer to make it watertight, so it grows. We checked every set for it. Length here tells you nothing.
Kristina: Any other giveaways you look for?
Stephen: Options that don't match the lead-in grammatically, "always" and "never" in distractors, and a stem that repeats a word from the right answer. If you can answer without knowing the medicine, the question has failed.
Kristina: Why British spellings and units?
Stephen: It is a UK licensing exam. You will see "haemoglobin" in g/L and glucose in mmol/L on the day, so that is what you should practise with.
Kristina: Guidelines change all the time. How do you know your answers are still right?
Stephen: Before writing any set, we checked the current version of every guideline it relied on. That matters more in primary care than almost anywhere, because so much of it is driven by NICE thresholds and pathways.
Kristina: Can you give some examples of things that had changed?
Stephen: Plenty, just in the last two years, asthma guidelines have changed, cancer guidelines, cervical screening, childhood immunisations … I could go on!
Kristina: What do you do when the guidance is uncertain?
Stephen: We don't test it. If an area is under review, or national bodies disagree, we write the question around something that is stable. It’s unfair to mark a student wrong on a point the experts haven't agreed.
Kristina: And when guidance changes after the questions are published?
Stephen: Every question is referenced, so we can find every item that depends on a guideline when it changes and revise them. If you spot one that looks out of date, please report it using our portal. Or if you just want to discuss the answers, go ahead!
Kristina: With so many questions, how do you catch mistakes?
Stephen: In two ways: automated checks on structure, and reading for content. The automated checks run on every question and confirm that there are exactly five options, in alphabetical order, the marked answer matches one of the options word for word, the lead-in is a single "What...?" question, every question has a full web link to its source, each set has the right mix of 5 Basic, 10 Medium and 5 Advanced. The content reading by experts is something though that no computer can do - is the scenario realistic, is the answer defensible, and would a GP recognise this patient?
Kristina: Do you use AI to write them?
Stephen: Some of our authors use AI as a drafting tool, and we’re open about that. It speeds up the first draft of stems and explanations. It doesn't decide what is tested, which guidance applies, or whether an answer is right. Those judgements, the guideline checks and the house style are ours as a panel of real human clinicians, and the drafts are checked against them before anything goes live.
Kristina: Does that make the questions less reliable?
Stephen: No, because the checking process is so rigorous. We haven’t detected any differences in the questions which are started off by AI or those fully written by clinicians. And every question points to its source. You don't have to trust us - you can open the guidance and see.
Kristina: Last question. What's the best way to use these questions?
Stephen: Three things. Read the explanation even when you get it right. Half the learning is in why the other four options are wrong. Work by presentation, not by disease. That is how patients arrive, and how the AKT is mapped. Finish a set, then look at which decision type tripped you up: diagnosis, investigation or management. Follow the links. If an explanation cites NICE guidance you haven't read, open it. The recommendations section is usually short, and it is exactly what the exam draws on.
Kristina: And if I think a question is wrong?
Stephen: Tell us. Students reporting problems is one of the best quality checks a question bank can have, and every report gets looked at. We usually respond within 12 hours. We’re very happy to have a debate about the rights and wrongs of each question - that’s why we all became educators after all!