Choose the correct statement!
What do you think is the correct answer?
Welcome to Bad Qs. A bar where your choice always matters. You may have seen this funny multiple-choice question used as an icebreaker at learning geek meetings. We will return to the solution at the end of this article. Now let’s look at a real example.
I came across an example of this while preparing for the AWS Certified Cloud Practitioner certification.
Which of the following options is correct regarding AWS Database Migration Services (AWS DSM)?
A) AWS DMS can migrate databases from on-premises to AWS
B) AWS DMS can migrate databases from AWS to on-premises
C) AWS DMS can migrate databases from EC2 to Amazon RDS
D) AWS DMS can have Amazon Redshift and Amazon DynamoDB as target databases.
E) All of the above
Since you may not know the answer to your question, let’s simplify this to a general dilemma.
Which of the following options is correct?
A) That’s true.
B) True
C) It’s true
D) true
E) All of the above
What is your choice? A, B, C, D, or E?
Problems with “all of the above”
If a multiple-choice question has 5 choices and you do not use “all of the above,” the theoretical chance of making the correct choice is 1 in 5 (20% chance). In reality, not all options are equal, so psychometrics has a variety of science-based approaches to determining more realistic values. 🙂
The moment you replace the last option with “all of the above” this becomes a much easier game. As long as we can identify two TRUE answers, we know that the expected answer is E. As long as you can identify one answer that you know is definitely FALSE, also eliminate option E.
But there’s a bigger problem to be aware of. If this test is a high-stakes test within your organization (certificate, legal, etc.) and you allow participants to challenge the questions, there are likely to be discussions such as: there is.
Choosing a TRUE answer is not the same as not choosing a FALSE answer.
What does that mean? It doesn’t matter if there is a better option, as long as the option you choose satisfies the gist of the question. It all depends on how you word the question, or how explicitly you instruct the user to choose the best option.
Which of the following options is correct?
A) That’s true.
B) True
C) It’s true
D) true
E) All of the above
For example, if you choose A, you can claim that the stem (“Which of the following options is TRUE?”) is completely satisfied by option A. Is option A TRUE? Yes. correct. I’m not saying that the other options are wrong, I’m saying that option A is the correct answer.
Lesson learned: Avoid using “all of the above” unless absolutely necessary. However, if you do so, you must write the question so that only one of the choices is correct, either by ranking the correct answers or by removing some of the choices.
For example, you might see question text that includes words such as “best” answer, “easiest,” “fastest,” or “cheapest.” Good wording suggests that more than one choice may satisfy the question statement. However, I’m looking for one of them in particular. Context matters!
If you really want more than one correct answer, use multiple choice. When it comes to multiple selection, there are two ways to indicate the number of options to select: explicit and implicit. You can explicitly say “Choose two.” Or just suggest that there is more than one correct answer by saying, “Select all that apply.”
There are significant differences between these two approaches.
Suppose you have 5 choices and you can choose 2 correct answers. If you explicitly ask them to choose two options, the challenge will be 2 out of 5. There are 10 ways to choose 2 out of 5.
If you say “Select all that apply”: 0 out of 5 + 1 out of 5 + 2 out of 5 + 3 out of 5 + 4 out of 5 + 5 out of 5. That’s 32 ways. Next, you need to consider whether to award partial points.
Back to the question of bad questions!
what was your answer? A, B, C, D, E, F, or none of them?
A) Incorrect – All options below are correct. For example, it conflicts with options E or F. Also, if A is true, then B is also true. And CF must be both true and false at the same time…
B) Incorrect – All options below are incorrect. For example, it conflicts with option D.
C) Incorrect – This is more interesting. Claim that either A or B is correct, but not both. This contradicts option A because option A states that both A and B are true. Also, option B states that C is wrong, so this contradicts B.
D) Incorrect – Option D is not correct because it also means that both A and B are correct at the same time. (Note: If you use “all of the above” or “none of the above,” don’t shuffle them. Order matters!)
E) Correct – Check this for yourself.
F) Incorrect – If option F is correct, then answer AD is all incorrect. However, if that is true, option E is equally true. Oops. And, as option F states, if option E is wrong, then AD cannot be entirely wrong.
conclusion
A) Don’t use “all of the above” or “none of the above” unless you really have to.
B) Use “None of the above” and if it is correct, make sure to tell the user the correct answer in the feedback.
C) all of the above
update
Since the original article, you can now use AI to support and write your questions. So let’s close Bad Qs Bar for good!Originally published on www.linkedin.com.
Source link
