The LSAT-India sit-at-home online exam for law school aspirants, which started on Sunday (19 July) and has been continuing with testing sessions throughout the last week, has almost completed its run until this Sunday (26 July). And although it’s apparently not been without any hitches, those hitches may be useful lessons that should be heeded by the other bigger upcoming Covid-era law admissions tests.
Indeed, many eyes have been on the LSAT-India, which had been the first entrance test to announce an online and remotely proctored exam, with NLU Delhi for its All India Law Entrance Test (AILET) also deciding to go the same way, while the other national law schools’ vice-chancellors’ consortium had rejected an online proctored model in favour of physical tests taking via a computer in test centres (with medical precautions).
The US Law School Admission Council (LSAC) administers the LSAT-India, and we asked its US-based vice president of emerging markets and business intelligence, Yusuf Abdul-Kareem, for a mid-mortem of what had gone right and what hadn’t.
Only around 5,500 candidates ended up sitting out of nearly 12,000 interested
We had reported in late May that 11,790 had applied for the exam (for free).
However, the number of students who eventually paid the exam fees and took the exam was much lower, at only 5,632 candidates.
Covid was likely partly to blame for the steep drop in numbers.
“Oftentimes we have like students that submit applications and don’t go through the process to pay this, so they don’t meet the deadline,” explained Abdul-Kareem. “We heard certain students had been granted admission into universities, before even taking any exam, and in some cases, students weren’t sure what they were going to do about law school. And others were waiting to do what the CLAT is going to do and AILET was going to do.”
“So while we are happy and delighted to have the number of students that we have to sit for the exam, we certainly are looking forward to working with our alliance members to grow that number,” he added.
Those numbers are dwarfed by the expected 60,000+ who have applied for the CLAT (apparently the consortium is expecting a record number this year) and also the AILET, which had 20,000 last year (as well as other law entrance tests, such as Delhi University’s for instance).
Split format
One unusual thing about the online LSAT-India this year versus the AILET and the Common Law Admission Test (CLAT) is that since Sunday, the LSAT had been conducting its online tests in three batches per day, of up to around 500 students each.
“The reason why we chose to do it this way, is just to make sure we don’t have broad disruptions,” said Abdul-Kareem. “Given the potential for connectivity issues, we felt like doing it over a series of timeslots a day, would allow us to troubleshoot and address any issues, that were identified in the first or second wave of tests.”
So are the first batches to take the test guinea pigs, we asked, playing devil’s advocate. “I wouldn’t say guinea pigs,” he said. “We arrived at this junction based on a lot of thoughts and thinking and experience...
“It is our best effort to ensure we have a quality testing.”
In light of the difficulties the LSAT-India would encounter on its first day, it turns out that running the test in batches was a good idea, if an exceedingly complex solution to implement in practice.
Teething troubles: On first day 14% couldn't continue
On the first day of exams, on Sunday, around 14% of the 1,000 candidates couldn’t actually sit the exam.
Surprisingly, perhaps, the issue was neither technical (though in some cases testing software had crashed computers) nor related to patchy internet connectivity, which many predict would cause mayhem for a national exams in some rural parts of the country.
The biggest problem actually boiled down to communication.
“The disruption that has challenged us the most, has little to do with the technical difficulty, and more to do with people following instructions, quite honestly,” he said. “The Herculean challenge, the lift, if you will, has really just been around trying to get students when they take their pictures [with their webcams before starting the test] to hold up their IDs [cards].”
But because of the staggered model, the LSAT was able to take fairly prompt action. “We noticed it during the first slot and immediately took action, to not only communicate with the students who were in that first slot, but also send out messages to all subsequent students, but also subsequent days, to ensure we don’t have that issue going forward.”
“If you have just one single point of failure - if we just had one single event - the ability to troubleshoot is limited in whatever way you administer the test,” he explained. “If you can’t successfully resolve that issue in real time, the students will suffer. Because we have done the exam in three slots a day, it allowed us to really make some changes for the better, and address any issues for the next round of students that are going to take the test.”
The effort was at least in part successful. “We did a lot of communication via text, via email, and we think, from that, students are getting the message now and are having a better check-in overall experience,” said Abdul-Kareem on Wednesday evening, noting that the percentage of students who did not show their identification cards alongside their mugshot, had decreased to only 2% over the preceding two-day period, which had seen 1,500 test takers (having completed 4,500 total exams at that point).
Those hundreds of students who had not shown IDs to the proctors, to prove that it was them taking the test, were not able to take the test and were re-scheduled to take it later in the week.
Which poses one very obvious question: how do you prevent cheating and the potential leaking of questions when having so many test slots?
How to run the same test over 8 days without hacking the bank
The answer is that LSAT-India has a big randomised bank of questions, as well as apparently some magic sauce.
Abdul-Kareem declined to confirm the exact size of LSAT-India’s question bank, when we asked whether a determined coaching centre could send fake candidates on day one to remember as many questions as possible out of the bank in order to leak them to later test takers.
However, he explained that in addition to a large bank of unique questions, “our psychometricians developed a testing model that would allow each individual test taker to get his or her unique test form so no candidates are going to get the same test form”, according to Abdul-Kareem.
In addition,
psychometrics (which is sometimes derided as a rather inexact science), would be used to spot any irregularities in the testing answers given by candidates.
In some way, one can imagine this model may have worked for the LSAT-India though it may be hard to scale up.
For an even more competitive exam such as the CLAT, let alone the IIT’s Joint Entrance Examination (JET), where there are huge cottage industries and money dedicated to extracting tiny advantages, there are obvious question marks whether such a system could withstand concerted attempts by professionals to break it.
But then again, in Covid-19 times, there are really only least bad options to conduct admissions test.
The C word and technology (Windows only)
Which brings us to the big potential issue on everyone’s mind in online proctored exams: the potential for cheating.
First of all, the software developed by LSAC partner and international testing multinational Pearson Vue, has a few tricks up its sleeve.
First of all, it was properly compatible only with Windows operating systems (OS), which means no dice for those on Macs (and probably none for
Linux or other niche OS users either, though Abdul-Kareem was not able to confirm that for sure).
The programme running on candidates computers, will then take complete control of the system, much like an anti-virus programme (or indeed, a virus) might, and shut down everything else running on the computer, so you are left with only the test window.
Images are captured via the computer’s webcam and the (AI) artificial intelligence system can apparently do things like tracking the eyes of candidates to detect suspicious activity.
“The computer will capture anything and everything that isn’t you and also your eye movement: we can tell if you have another screen or looking down or looking at other [materials],” said Abdul-Kareem (though scratch paper is allowed).
So have any cheaters been caught?
“We have had wind of suspicious behaviour and the AI system has flagged some activity,” said Abdul-Kareem, “but we have not cancelled anyone’s scores yet because we are doing an investigation.”
“I would say that there were maybe one or two students that the system has flagged thus far,” he added. “It hasn’t been rampant and hasn’t been pervasive.
“We don’t feel that the security and integrity has been compromised in any shape or form.”
That said, results have not been evaluated yet and trying to catch any potential cheaters would be ongoing, based on a “rigorous psychometric review... that will unearth any suspicious” patterns, and other analysis of the answer patterns, scores and proctoring video footage, according to the LSAC India head.
If anyone is conclusively identified to have cheated, that would be against the law, added Abdul-Kareem, including potential civil or criminal action. “When we discover that people were breaking the law and [cheating or] pirating content, our IP, for instance, we will definitely pursue any and all legal avenue to prevent it.”
Advice for CLAT, AILET, others
We asked if he had any potential advice or lessons learned for the CLAT and the AILET, which will only hold their exams late next month.
Number 1, according to Abdul-Kareem: “Develop a clear communication channel with students and over communicate... across multiple channels..., whether it be email, WhatsApp, Telegram, Facebook, Instagram, Twitter.”
“I would also say to speak to others, whether testing organisations or institutions that have done remote proctored exam before, so you have the benefit of the learning and experience,” he noted. “One thing that helped us tremendously, was the fact that the LSAC administered LSAT-Flex in the US.”
“The other advice I would give, is to make sure you have a robust customer service and troubleshooting support line and support system.”
The staggered exam - of now more than 500 candidates at once - would have helped LSAT deal with any issues in real time. But imagine how you would deal with it if just 1% of 60,000 students taking the CLAT were to have a technical issue on exam day with signing into the exam or the software working.
“It’s really about how you scale,” said Abdul-Kareem. “Do you have requisite number of customer support to student ratio?”
threads most popular
thread most upvoted
comment newest
first oldest
first
But the biggest issue is this: less than 50% who had originally applied finally sat for the exam. Now you may blame Covid all you want to for that. But it finally boils down to one of two possibilities. Either less than 50% actually had the necessary infra to appear for online exams from home, thus vindicating CLAT's stance about accessibility. Or the rest simply do not consider places like Jindal a viable alternative to NLUs yet, just a fall back option (which is why they want to wait for CLAT, AILET etc.) Either way, it doesn't look very good. That's the unbiased version.
Jindal has inturn went around the back of LSAT and started taking JSAT for free. Hence there was no reason left to give LSAT.
Tl;Dr
JSAT introduced by Jindal for free makes the expensive LSAT redundant.
threads most popular
thread most upvoted
comment newest
first oldest
first