Following the NEET fiasco, resulting in the cancellation of UGC-NET exams, the removal of National Testing Agency (NTA) chief, and the CBI investigation, the Union education ministry has formed a committee to initiate a reform of the examination process and improve data security protocols. The Indian media, meanwhile, has been abuzz with statements about corruption.
The shockwaves that followed have led to policy recommendations on the desirability of integrating the national level and state level efforts. The AIDM article, National Testing Agency Yes, But Ensure State Level Participation and Social Justice, by Professor Rajan Gurukkal, is an example from a socio-political angle.
Corruption is not a novel problem in India. All of us, including the aam aadmi talk about it. Occasionally, some who have been proven to be guilty of corruption have been punished. But, as far as I can see, there has been little attempt to reform the dysfunctional systems that prompt corruption, and to deter them from it.
While corruption is a serious problem, more serious is the problem of poor design of exam and test questions. It results in considerable harm to the cognitive growth of learners, and injustice to those students who are highly capable, but are not good at what these questions probe into.
To understand the nature of this problem, we need to discuss the widespread use of Multiple Choice Questions (MCQs) in Intelligence Tests, Aptitude Tests, Entrance Tests, and Final exams.
Poor design of MCQs
The design of MCQs involves both their format and the kinds of potentials or attainments they implicitly seek to measure.The format of exams and tests range from those that expect essay-type answers to be completed in half an hour, to MCQs that need to be completed in one or two minutes.
At the end of the twentieth century, IIT entrance tests moved from the essay-type and short-answer questions to MCQs. Subsequently, the format spread to other entrance tests and final exams. According the article,Why are all online tests going MCQ-based and how does it fail holistic assessment? (The Indian Express,7 July 2024), the launch of the NTA has resulted in a shift to Computer Based Tests (CBT) with the MCQ format.
The use of MCQs in psychometrics is only a hundred years old. Frederick J. Kelly, the father of MCQs and the author of the Kansas Silent Reading Test, devised this format in 1914 to eliminate subjectivity. More recently, with the emergence of computational technology, MCQs became the most favoured format because it saves examiners from the drudgery of grading.
While MCQs eliminate the subjectivity of the examiners, and computer gradable MCQs save human labour, they focus entirely on lower level mental abilities, and fail to probe into thinking and deeper understanding. As a result, the use of MCQs in our tests and exams has an undesirable effect on what students invest time and effort to learn.
Why do I say so? To answer that question, we need to examine what potentials or attainments of learners that these MCQs allow us to measure.
The standard format of MCQs provides four or five options with one 'correct' option, and allows the candidates an average of one to two minutes to tick or click on the correct option. This format is appropriate for testing recall of what is memorized, and the mechanical skills that call for high-speed application, as in the case of calculations that can be performed by the algorithms built into a pocket calculator, or slightly more sophisticated algorithms like ChatGPT.
This format is ill-equipped to probe into reasoning abilities, critical reading, critical thinking, inquiry, conceptual understanding, and so on, let alone such things as imagination, insight, intuition, and integration of knowledge.
To use the terminology of the Nobel Laureate Daniel Kahneman, the existing design of MCQs tests fast thinking or type 1 thinking – the mechanical skills that can be acquired through extensive supervised practice. Strictly speaking, these should be called acquired knee-jerk reactions without thinking.
What they fail to test are what Kahneman calls slow thinking or type 2 thinking. These are the strands of what NEP 2020 calls Higher Order Cognition. We at ThinQ call it Academic Cognition, what is needed for thinking like a mathematician, like a scientist, like a philosopher, like a historian, and so on.
As early as 2010, in the article, Entrance Examinations for Science and Technology (Current Science Vol 99 No 10), I had presented an extensive critique of the design of assessment that uses MCQs. That critique was followed up in the NAAC White Paper 2022, Reimagining Assessment and Accreditation in Higher Education. Yet, the poor design problem, unlike the corruption problem, has attracted very little attention from the stakeholders of institutionalised education: the ministry, the industry, expert committees, education administrators, students, their parents, and the public.
An alternative: Enhanced MCQs
In my article mentioned above, I had suggested the use of a different format that I called Enhanced MCQs (e-MCQ). Unlike the standard format, e-MCQ provides between ten and twenty options to pick, where different options have different positive or negative marks, and candidates have to pick two or more options to get full marks for the question. Not knowing the number of options for full marks, combined with negative marking, prevents or at least strongly discourages random guessing.
A three-hour test using the standard format typically carries a hundred questions. The Teacher Eligibility Tests are worse: these tests require candidates to answer a hundred questions in two hours. Such tests ensure the absolute absence of thinking while answering questions.
Unlike such tests, each question in an e-MCQ test requires students to spend twenty to thirty minutes to read a short text and think about the options before answering the questions. This means that such a two-hour test cannot have more than four or five questions. The basic idea is that at least about eighty per cent of the time should be allocated to reading and thinking. These are the closest to what are called 'open-ended assessment tasks' that do not assume a single 'correct' answer.
Examples of learning-assessment tasks
To get a sense of the learning outcomes that assessment tasks aim to test, let us consider a few examples of tasks. These can serve as both learning tasks that facilitate HOCCs (Higher Order Cognitive Capacities), and assessment tasks that test those HOCCS. Furthermore, while they are not formulated as e-MCQs, they are amenable to being converted into that format. My purpose here is to highlight the nature of the learning outcomes that the questions facilitate as well as test.
Reasoning from Definitions
1. Consider the following definitions of triangles
Def. (i) A triangle is a shape with exactlyt hree vertices connected through straight lines and only straight lines.
Def. (ii) A triangle a shape with exactly three non-colinear vertices connected through straight lines and only straight lines.
Task 1: Pick one of these definitions to prove that straight-angled triangles do not exist.
Task 2: Now pick the other definition to prove that straight-angled triangles do exist.
2. Consider the following definitions of democracy
Def. (i) Democracy is a system in which all those who are affected by a decision have an opportunity to influence the decision.
Def. (ii) Democracy is a system in which people elect their rulers through voting.
Task 1: Pick one of these definitions to argue that democracy exists in many parts of the world.
Task 2: Now pick the other definition to prove that democracy does not exist anywhere in the world.
Unearthing Implicit Assumptions
Consider the following conversations:
Conversation (i)
Sumi: Zeno is the best teacher in our college.
Plato: Why do you think so?
Sumi: Because he has been getting the highest ranking in the student feedback scores.
Conversation (ii)
Plato: Socrates is immoral.
Sumi: Why do you think so?
Plato: Because he kills rabbits.
Task: In providing rational support for their judgments, Sumi and Plato make certain assumptions. Identify those implicit (unstated) assumptions, and state them explicitly..
Critical Thinking
Newton’s second law of motion says that for a body whose mass is constant,
f = m.a
Consider ball A moving with constant velocity (zero acceleration), hitting a ball B at rest (zero acceleration). Given that a is zero, m.a is zero. Hence, f is zero. This means that neither of the balls is exerting force on the other. So ball A will continue to move with the same constant velocity, and ball B will continue to be at rest.
You are left with the following options:
Option 1: Newton's equation is wrong.
Option 2: Newton's equation is not wrong, but the interpretation of the example given above is wrong.
Which of these options would you choose? State your reasons.
To get a sharper sense of what these questions probe into, it might be useful to compare the above with the familiar type of questions:
Question 1: Prove that the sum of angles in a triangle is 180 degrees.
Question 2: A projectile with an initial speed of 10 meters per second is shot from location A at an angle of 30 degrees, and lands at location B. Calculate the distance from A to B.
Question 3: What is the molecular formula for glucose?
Question 4: Discuss the mechanisms of Evolution.
Question 5: What are the causes of the Second World War?
Unlike the previous set of questions, these questions test recall of memorized information, and skills of mechanical application.
Some of us have used e-MCQs in our own tests and exams, and found them to be ideal for probing into Higher Order Cognitive Capacities (HOCCs), including critical reading, critical thinking, reasoning, and inquiry.
For more examples of tasks that can be used for assessment of HOCCs, the reader may take a look at the draft of the textbook Foundations of Knowledge and Inquiry across Disciplines.
Concluding remarks
In an educational culture such as ours, school education is a means to admission to a college, and college education is a means to get employment that ensures a good income. We also assume that the higher the income, the greater the happiness. Regardless of the legitimacy of these assumptions, we must acknowledge that they are part of our current society.
It is natural therefore that doing well in school exams, entrance tests, aptitude tests and college exams is the primary driver of learning for our students. And helping them to achieve their aspiration is an important responsibility of educational institutions and their teachers.
It is imperative therefore that we solve the design problem in our tests and exams, such that they become drivers for high quality learning. Without that, our system of education will continue to deteriorate, regardless of how successful the national level initiatives are in solving the corruption problem and improving data security protocols.
(KP Mohanan is a former professor at IISER Pune, a founding member of ThinQ (www.thinq.education), and an Honorary Advisor to Kerala State Higher Education Council. He got his PhD from Massachusetts institute of Technology in 1982, and taught at MIT,University of Texas at Austin, Stanford, and the National University of Singapore, before returning to India in 2011 to join IISER Pune.)