Frequently asked questions about progress testing

What is a Progress Test?

The Progress Test is an objective measurement to test students’ knowledge and progression. The medical faculties of Nijmegen, Maastricht, Groningen, Leiden, Amsterdam (VU and AMC) and Rotterdam work together to provide their students with relevant medical questions of high quality. During the Progress Test, it’s not the medical curriculum that is tested, but rather the broad knowledge that students picked up during their medical education. The Progress Test assesses the student’s readily available knowledge. Up until now, the Progress Test is made on paper or digitally by all medical students of the aforementioned faculties. The difference in knowledge levels between students is expressed as a test score.

In March my progress test result was Insufficient, but the standard for my cohort was lower at the previous December test; can my score be upgraded to Sufficient?

No that is not possible. Standards are calculated separately for each progress tests (see information about standards) and the standard of the December test applies exclusively to that test. It cannot be used to calculate outcomes on any other test.

The score I calculated myself differs from the official outcome of the progress test. What should I do?

You should first check whether you have included all key changes and cancelled questions when you calculated your score. Cancelled items are excluded when the percentage score is calculated (so if two questions are cancelled, you should divide by 198 instead of 200). If you still find a difference between your score and the official outcome, you should ask to look at your original test form. It is very likely that you will find differences between the original form and the copy you kept.

Initially, my test result was Sufficient, but after key changes and cancelled questions it is no longer so. Is there any way I can get a Sufficient result after all?

The definitive test results are determined after all key changes and cancelled questions have been processed, because at that point all errors have been removed from the test and the score can be determined accurately.
So, with a test of 200 items you will have a certain initial score, which may change due to key changes and cancelled questions. The true, definitive score is the score calculated after all key changes and cancelled questions have been processed, because the initial score was based on errors in keys and faulty questions.

A key change can be advantageous to some students (their score changes from negative to positive) and disadvantageous to other students (a change from negative to positive). That is only logical, there was a technical error, such as a typing error in the correct answer when the test was prepared.
When a question is flawed (for example: both answer options are correct, or the wording was wrong) it is cancelled. That means that all plus and minus points scored on that question are cancelled as well. This is logical too. When a question is cancelled, 1 is detracted from the number of questions in the formula used to calculate the percentage score: for example: 200 becomes 199, and the percentage score is calculated by dividing the absolute score by 1.99 instead of by 2.

Some progress tests may turn out to be more difficult than others. Is it really possible to monitor your own knowledge development?

If you look at a series of PGT results, the development of knowledge shows clearly, both for groups and for individual students. It is true that PGTs can vary in difficulty, and this may lead to a bend in the line representing the (group or individual) Correct minus Incorrect scores as well as changes in test standards (since these are norm referenced). Nevertheless it is quite possible to measure progress over a longer period of time, because a series of tests shows a reliable tendency of knowledge development (see for instance Van der Vleuten CPM and Driessen EW. Toetsing in probleemgestuurd onderwijs, Hoger Onderwijs Praktijk, Wolters-Noordhoff, Groningen 2000, 9. 27-32).

Moreover, you can see how your score compares with the score of your reference group. This comparison indicates if your score remains constant, improves or deteriorates compared to the group average, irrespective of the difficulty of the test.

What is adaptive testing?

(Computer) adaptive testing provides students with a series of questions that is adapted to the student’s knowledge level. Since the difficulty of all questions is known, every student can receive a Progress Test that fits his or her knowledge. Such an individualized, automatically generated test has the benefit that it does not have to be made by all students at the exact same time.

How does computer adaptive testing work?

With computer adaptive testing, the computer selects questions from a large question bank. The selection of every subsequent question is based on the answer given to the prior question. For example, if the student answers a question correctly, the algorithm will select a slightly more difficult following question from the question bank. This process is maintained until the student gives an incorrect answer, which will result in the algorithm selecting a question that is easier than the preceding question. This performance-based selection of questions is sustained until the system has gathered adequate information to correctly estimate the student’s knowledge in certain disciplines. However, the blueprint (e.g., the Progress Test content specifications; categories and disciplines) of the Progress Test will not change compared to the ‘original’ version. The algorithm will still cover all disciplines.

The principle of adaptive testing is illustrated in the following figure:

Adaptive Testing principle

What are the benefits of adaptive testing?

Computer adaptive testing ensures a completely individualized test. The test is thus different for every student. Computer adaptive testing is a state-of-the-art technology that enables precise estimation of students’ knowledge in a short period of time. The number of questions that needs to be answered for this precise estimation is lower than before; only 100-200 questions are required.

Since all students will make an individual and therefore different test, the Progress Test will not have to be made by all students at the same time. In this way, education and clinical activities do not have to be interrupted in order for the Progress Test to take place. Additionally, the implementation of the computer-adaptive Progress Test enables the use of other media, including sound and video, for questions in the future.

In short, the most important benefits of adaptive testing are the flexibility that it creates and its precise measurements.

What changes about the questions with adaptive testing?

The new approach of the Progress Test offers many benefits. However, some changes were necessary. Firstly, students can no longer take the test booklet with questions home, since the adaptive Progress Test will now be held exclusively online. The questions will then remain in the question bank. To ensure that students still remember what the questions of the Progress Test were about, the students will receive a short description of the content of the question, as well as whether or not the question was answered correctly. Secondly, the option for a question mark will be removed. This feature does not support the adaptive environment, because it will not provide adequate information whether or not the student has knowledge on the topic of the question. It is therefore important that you answer every question as good as possible, as this determines the next question that you will receive. However, the adaptive characteristic will eventually improve the Progress Test, resulting in less questions that you will not be able to answer.

What changes about the testing with adaptive testing?

The adaptive Progress Test will only be held digitally (on a computer). This approach has been tested for several years and has already been implemented in the international medical curriculum of Maastricht. From September 2022 onwards, the Progress Test will not be held in its original form anymore. Every student will now make the Progress Test digitally during an assigned time slot on campus. However, this time slot will not be the same for every student, since every student will make a different, individualized test.

The adaptive Progress Test will be shorter than the original Progress Test due to computer adaptive testing technology. You can watch the following short video on the technology behind computer adaptive testing: https://www.youtube.com/watch?v=ZvFNwR8ABo4&t=65s, or search YouTube for “computer adaptive testing” to find a video that suits you best.

Why did the Progress Test approach change to adaptive testing?

The current Progress Test is a very long test consisting of 200 questions, with a maximum time of 4 hours. The test is based on the level of a basic medical doctor (‘basisarts’), which means that many students (especially in their bachelor’s) have difficulties answering the questions. Furthermore, the fact that all students have to make the original Progress Test at the exact same time forms a challenge logistically.

The adaptive Progress Test will be shorter and will offer questions on the level of the student. Adaptive testing has also been shown to be more efficient since students require less time to finish their test. The test score will not only be more informative, but will also correspond better to the estimated knowledge level. Additionally, the reliability of the test score is comparable to the test score of a two times as long paper Progress Test.

The adaptive Progress Test will not be held at one time point, but all students will be distributed over time slots during one week. The Progress Test will still be held four times a year.

What will not change with adaptive testing?

The Progress Test will still be held four times per year. The rules for determining the end result at the end of the year will still apply.

What kind of questions will be asked during an adaptive Progress Test?

Computer-adaptive testing is only possible if the difficulty of a question is known. This difficulty can only be estimated by asking the question. There is a big pool of questions available of which the difficulty is known, because they have been asked during previous Progress Tests. However, this pool of questions has to be updated regularly, meaning that also newly developed question will be asked during the adaptive Progress Test. Because the difficulty of these novel questions is not known, these questions will not count for the test score. The results will however be used to calculate the difficulty of the novel question in order to add the novel question to the question bank. We estimate that approximately 20% of the questions in the Progress Test will be novel.

Who will encounter the Progress Test changes with adaptive testing?

All medical students studying at the medical faculties of Maastricht, Nijmegen, Groningen, Leiden, Amsterdam (VU and UvA) and Rotterdam will make the new adaptive Progress Test from September 2022 onwards.

What is different about the test score and feedback with adaptive testing?

After finishing the Progress Test, students will receive (following the determination of the standard/norm) a score in TestVision. Moreover, students will receive feedback on their progression in the Prof system. This will thus all remain the same.

Two things that will change, are:

  • The score will be shown on a different scale. It is necessary to express the score on a standardized scale, since every student will make a different test. More information on standardized scaling can be found at:

https://www.youtube.com/watch?v=2JjaWQZChqs

The scale will be set up in a way that fits the scores that you’re used to as good as possible. The Prof system will provide graphical information on your performance relative to the entire cohort. Research has shown that properly analysing the feedback in Prof results in better performance.

  • You will receive a report of all correctly and incorrectly answered questions in TestVision. All questions in the question bank will get a short question topic. You can use this question topic as an indication what to study.