Progress Test: Frequently Asked Questions

What is a Progress Test?

The Progress Test is an objective assessment designed to measure a student’s medical knowledge and its development over time. Since the 2022–2023 academic year, all eight Dutch medical schools have collaborated on the interuniversity Progress Test to provide students with relevant, high-quality questions. The test does not assess the curriculum itself but rather the knowledge students acquire throughout their medical education. It consists of 135 multiple-choice questions that assess the application of factual medical knowledge.

What is Computer Adaptive Testing (CAT)?

Since the 2022–2023 academic year, the interuniversity medical Progress Test has used Computer Adaptive Testing (CAT). In an adaptive test, students are presented with a sequence of questions. As the test progresses, the computer selects questions from a large question bank that increasingly match the student’s estimated level of knowledge. Because the difficulty of each question is known in advance, every student receives a personalized test tailored to their ability level. Consequently, individualized tests do not need to be administered simultaneously to all students.

In CAT, the computer selects questions from a large item bank. Each subsequent question is based on the student’s performance on previous questions. For example, if you answer a question correctly, the algorithm selects a slightly more difficult question. This continues until you begin answering questions incorrectly, after which the algorithm presents somewhat easier questions. This performance-based selection of questions is sustained until the system has gathered adequate information to correctly estimate the student’s knowledge in certain disciplines.

The test blueprint (content specifications) remains unchanged; the algorithm simply ensures that all blueprint domains are adequately covered while adapting question difficulty to your performance.

The principle of adaptive testing is illustrated in the following figure:

Considering that a student does not perform equally well in every discipline, a practical example will look like this:

The horizontal line is the student’s predicted knowledge level at the end of the Progress Test. The other, more wavy line between the ticks shows the knowledge level determined by the computer during the Progress Test, which was used to determine the difficulty level of the following question.

What are the benefits of adaptive testing?

Computer adaptive testing ensures a completely individualized test. The test is thus different for every student. Computer adaptive testing is a state-of-the-art technology that enables precise estimation of students’ knowledge in a short period of time. The total number of questions during the Progress Test is 135.

Since all students will make an individual and therefore different test, the Progress Test will not have to be made by all students at the same time. In this way, education and clinical activities do not have to be interrupted in order for the Progress Test to take place. Additionally, the implementation of the computer-adaptive Progress Test enables the use of other media, including sound and video, for questions in the future.

In short, the most important benefits of adaptive testing are the flexibility that it creates and its precise measurements.

The technology behind adaptive testing is explained in the following video: https://www.youtube.com/watch?v=ZvFNwR8ABo4&t=65s

Types of questions

Because question difficulty adapts to your ability, you are less likely to encounter questions that are either extremely easy or impossibly difficult. During the test, you may therefore feel that many questions are challenging, as they are presented around the upper limit of your current knowledge.

Since each answer determines which question comes next:

• You cannot return to previous questions.
• You cannot change earlier answers.
• You cannot skip questions and return to them later.

If you want to check the type of questions that are presented during a Progress Test, please visit the iVTG website (www.ivtg.nl), which offers a complete Progress Test of 200 questions. Additionally, the question bank is made available (to a certain extend) in collaboration with Medisch Contact through the website “Arts in spe”, to be found at www.medischcontact.nl/kennis-carriere/voortgangstoets.htm.

 

What kind of questions will be asked during an adaptive Progress Test?

Adaptive testing is only possible when the difficulty level of each question is known. This information is obtained from previous administrations of the question.

The question bank contains a large number of previously used questions with established difficulty levels. However, the bank is continually updated with newly developed questions. Since the difficulty of these new questions has not yet been established, they do not contribute to your final score.

Instead, your responses to these questions are used to calculate their difficulty for future use. Each 135-question test includes 15 newly developed “pretest” questions distributed throughout the exam.

After completing the test, the iProF feedback system indicates which of your questions were pretest items that did not contribute to your score.

Scoring and feedback

After completing the examination, you receive your results in TestVision. Later, detailed feedback on your progress becomes available through the ProF system. The feedback includes graphical comparisons of your performance relative to the entire student population.

Because the difficulty of each question has already been established, the passing standard is fixed rather than adjusted based on the group’s performance. In other words, an absolute standard is used instead of relative grading.

Your performance is expressed on a standardized scale. More information on standardized scaling can be found at: https://www.youtube.com/watch?v=2JjaWQZChqs

TestVision also provides an overview showing which questions you answered correctly and incorrectly. Every question is accompanied by a brief topic description that can help guide further study.

How is the difficulty of novel questions determined?

The level of new questions is determined by the so-called Rasch model using the item-response theory. This means that the chance that one student answers a question correctly is compared to the chance that this student answers the other questions of the test correctly (the knowledge level of the student). These two aspects can be plotted to determine the difficulty of the novel question with respect to the “average question”. The figure below illustrates this principle. The vertical, black line is depicted just right of the 0 line, which indicates that this question is a bit more difficult than the average.

How many questions are currently in the question bank?

Approximately 7800 questions.

When a question is answered incorrectly, how often will questions be asked from the same discipline?

The result of an individual question does not affect the choice of discipline of the subsequent question.

My test featured multiple questions on the same subject, is that supposed to happen?

If you have difficulty with a particular subject, multiple questions on this exact subject might be more noticeable. The consequences of this are presumed to be small, considering the limited impact of two questions on your final results.

There are two explanations for two or three questions on the same subject:

1. In a certain section of the question bank there could be multiple very similar questions and two or more could be given to you by chance. As the level of difficulty is adjusted along the way, based on your score in previous questions, a questions answered wrong in the begin might return later on in a simpler format, making it easier to answer.
2. One of the questions could count towards your final result while the other is a pretest item and does therefore not count. How does that work? Not all questions in your test contribute to your final result. Dispersed throughout the test are 15 so-called pretest items. These are inserted in the test to assess their difficulty level for inclusion in the question bank and use in future progress tests. In the selection of these pretest items, the content of the previous questions is not taken into account.

Although having double questions does not have major repercussions and could discourage studying, the progress test commission aims not ask multiple questions on the same subject. For that reason all questions (ca. 7000) and all new questions are checked additionally for textual similarities to then be compared manually for comparability in questioning. The questions that are considered too similar are marked as ‘enemy items’; TestVision then automatically prevents these questions from appearing together in a student’s test.

To identify these enemy items, your input is very helpful. If you find two or more (nearly) identical questions, please report which questions this concerns, either during the test, directly afterwards or during the insight opportunity. This report can be filed in TestVision’s commentary box. Please mention the question numbers and the subject so we can process the report properly.

Comments and test review

Since each Progress Test contains thousands of possible questions, comments submitted through TestVision are analyzed primarily for newly developed pretest questions. Existing questions are mainly monitored through changes in their psychometric characteristics, which may indicate issues with content validity.

Students who wish to exercise their legal right to review their completed examination may do so through their faculty. Registered students are assigned a 45-minute review session conducted under examination security conditions.

Which possibilities are offered for students with a disability (e.g., having difficulties with digital testing)?

This will be managed by the local exam committee/examiner.

How long will it take for students to receive their test result?

This depends on the time slot of your Progress Test and when the last Progress Test will be held during that week. The maximum amount of time to receive the test result is established in the local study- and exam regulations (Dutch: “opleiding- en examenreglement” or OER).

How is the result of the test determined?

Student scores are calculated using the Weighted Likelihood Estimation (WLE) method developed by Thomas A. Warm (1989). The resulting score is compared with the average performance of the total student population and expressed as a Z-score. This Z-score is subsequently transformed to a standardized scale with:

• Mean = 35
• Standard deviation = 15

These values are based on the average interuniversity Progress Test scores from previous years.

Negative scores are possible because Z-scores themselves can be negative. For example, a student performing three standard deviations below the overall average may receive a standardized score of approximately –10.

Since CAT uses questions with precisely calibrated difficulty levels, a standardized scoring system is essential to ensure fair and consistent assessment.