While the National Law Aptitude Test (NLAT) for admission to NLSIU Bangalore’s LLB and LLM programmes was launched yesterday and NLS has announced a fully-proctored simulation to take place tomorrow, with the actual exam on Saturday (12 September), serious questions remain about whether the university will catch any of the inevitable cheating attempts (besides the plethora other issues, some of which have been highlighted in our Claxit live blog).

We’ve had the opportunity to play around with the mock exam and its technology and have collated the accounts and ideas of several candidates and other experts we have spoken to about cheating in online exams. For obvious reasons, all wanted to remain anonymous.

From this, as well as other data available, it is a certainty that some candidates, potentially hundreds, may try to cheat.

More importantly therefore, we examine whether there’s a realistic chance for NLS and its vendor to catch all those cheating attempts.

Spoiler: right now, the outlook is not great.

Disclaimer: This is purely for informational purposes

For the avoidance of doubt, this is not a guide to cheating: we are publishing this purely for informational purposes and are not inducing anyone to try and cheat the system.

First, cheating on a competitive exam is plain wrong. Second, apart from potential disqualification, it may ruin your education, career, and might result in criminal charges in the worst case. And third, it is probably a far more stressful and ultimately less successful experience for most than having prepared properly and taking the exam normally.

Nevertheless, the ideas and concerns raised in this article have been suggested from many of those who are going to take the NLAT, most of whom probably don’t intend on cheating themselves, as well as those who have taken other proctored exams. None of the below are rocket science or rely on information that is not in the public domain, and all fall within things that are achievable by unethical but enterprising students, let alone skilled / bored techies, hackers or makers.

The reality is that unless NLS and its vendor have confidence that they can stop attempts such as the ones described below, the integrity of the exam can not be guaranteed by simply not talking about such techniques. Security through obscurity is usually judged by experts as barely better than no security at all, so pretending the information is not out there or does not exist, won’t help out anyone except those who are intending to cheat.

“Can you 100% guarantee that cheating will be impossible in the NLAT without getting caught?” we have repeatedly asked NLSIU vice-chancellor (VC) Prof Sudhir Krishnaswamy and other members of the administration since Monday (7 September).

“Given the elaborate schemes people go through to cheat in offline competitive exams, is there not a significant risk that in a home-proctored test the risk of inventive schemes that will be impossible to detect via software and a webcam (particularly those with the means to employ or take the aid of third parties off-camera),” we added.

We received neither response or comment to that question.

A numbers game: Why it is certain some candidates will try to cheat

It is a near certainty that at least some candidates will try to cheat on the NLAT.

Even offline centre-based tests are clearly not impossible to cheat on - exhibit Vyapam scam - online proctored exams are potentially a gamechanger for candidates who seriously put their mind to breaking them, despite mitigations and fancy technology buzzwords like artificial intelligence and machine learning (which are often also used as snake-oil in the IT industry).

First out of the gate from 19 to 26 July had been the LSAT-India, of which JGLS Sonepat is the primary recipient of candidates.

Then, on 27 July, we had reported on accounts of cheating in the Symbiosis University admissions test, the Symbiosis Law Admissions Test (SLAT). One candidate had recorded his entire half-hearted / experimental cheating attempt on video and uploaded it on YouTube. He was not caught by the live proctoring though was apparently promptly identified and disqualified (and apparently forced to publicly recant and apologise on YouTube).

Only two days after our story, possibly coincidentally, NLU Delhi had abandoned its firm plan of a home-proctored proctored exam.

If there was ever a good time and good place for proctored exams, that may be neither now nor for NLSIU Bangalore.

34-200 were flagged as suspicious in LSAT/SLAT

We understand from authoritative sources close to each respective test, that the proctoring software for the LSAT-India and SLAT respectively flagged around 34 and 200 candidates attempts as “suspicious”.

For LSAT-India that works out to around 0.7% of its 5,500 candidates; for the SLAT, it is around 1% of its 20,000 candidates who did something that the system identified as suspicious.

The advantage that both of those tests had was that at that time, proctored testing was still a relative novelty for law admissions and very few people in India had taken one (and tested the limits of the sophistication of the technology).

That novelty has now worn off and on top of that, this is NLSIU we are talking about: the self-styled best law school in the country, as also quasi-rubber-stamped by the National Institutional Ranking Framework (NIRF) year-by-year. Competition will therefore be much more intense for a much smaller number of places and by a much larger number of candidates.

At this point, no one knows how big the NLAT really is except NLAT. Our repeated requests for a confirmation on numbers have remained unanswered.

But considering there are 70,000 takers for the CLAT, and 20,000 for Symbiosis’ SLAT, and having spoken to CLAT coaches, we would guesstimate that NLS may see upwards of 50,000 takers (with possibly the sky as a limit: with low application fees starting at Rs 125, who is to say some kids bored with nothing to do during lockdown, who may not have done well on the IITs entrance tests for example, won’t just sign up to NLAT and give it a shot for the heck of it? After all, it’s online and only takes 45 minutes.)

Update 11 September 2020: There are 30,298 candidates currently registered, according to NLS senior counsel Sajan Poovayya in the Jharkhand high court yesterday. Thanks to the commenter below for spotting this.

NLSIU is a prize for any candidate. And those who thought they may not have had a shot at NLSIU in the CLAT other than via a fluke a la Slumdog Millionaire questions, may feel they have nothing to lose by cheating on the NLAT.

Furthermore, considering the strained relations between NLAT and the CLAT, they might not even face disqualification from the latter, nor are they likely to face criminal sanctions except in the most egregious cases. Plus, since all the evidence that exists is basically low-resolution 640x480 webcam footage, there will probably be plausible deniability for ‘suspicious’ activity (which is why the LSAT, for instance, avoids labelling such activity as ‘cheating’ outright, because it can be impossible to know).

With one proviso (mentioned below in ‘Saving grace’), purely by the LSAT and SLAT numbers, NLS might therefore expect to deal with at least 210, 300 or possibly more attempts at cheating (providing we assume that the LSAT and SLAT proctors managed to catch everyone).

The below assessment of risks is therefore not merely theoretical.

First of many mysteries revealed: The vendor

First off, a mystery “vendor” with “experience conducting large-scale examinations in similar formats” is mentioned five times in NLSIU’s FAQs, in answer to frequently asked queries surrounding the home-proctoring and examination software. However, that “vendor” has never been named.

We have repeatedly reached out to Krishnaswamy and other members of the administration for comment since the NLAT was announced on 3 September but all have declined to confirm the vendor.

NLSIU also declined to confirm the name or further details of proctoring to Medianama in an interview published earlier today.

But that cat may be out of the bag since yesterday’s mock test, which mentions Aon for a fraction of a second when loading the test, which itself is hosted on the domain cocubes.com.

Aon company has done Sharda U admissions, worked with LPU, Jain U, KLU

Cocubes.com turns out to be CoCubes Technologies Pvt Ltd, an Indian company, which had been acquired by US-headquartered Fortune 500 human resources (HR) services giant Aon Hewitt, in 2016 when it was described as a “hiring assessment firm” with 110 employees.

Other than doing online assessments to screen candidates for jobs, there is no mention of it having been active in the examination or education space at that time. Nowadays, as an Aon company (which rebranded to Aon Solutions in June 2020), the title of Cocubes’ homepage now encourages visitors to: “Drive your talent strategy with online assessment solution”. Besides helping in hiring, it also offers online tests to help “institutes measure and improve employability of their students”.

Online exams seem to be a relatively newer vertical for Cocubes, and in Covid-times of little hiring of talent of taking place, potentially a more lucrative one. Cocubes claims on the exams microsite to have conducted “80000+ online exams successfully... with our partner institutes”.

According to its testimonials on its website, previous clients and work of Aon (of which Cocubes is now a part) have included Sharda University’s online admissions test since 2018, as well as Jain University’s online exam for MBA students, Lovely Professional University’s home proctored exams since 2017 (not clear if an admissions test), and KL University in an unspecified exams-related capacity.

We have reached out to Cocubes and NLS for further comment.

But this is no ITES Horizon Pvt Ltd

Nevertheless, the Aon brand name and the fact that Cocubes has done proctored admissions test before, should give at least a modicum of confidence that we’re not dealing with the kind of fly-by-night shell company operator opaquely selected by the, albeit much simpler, All India Bar Exam (AIBE) for the Bar Council of India (BCI).

However, big company does not equal foolproof, especially when we’re talking IT projects delivered on short timelines and particularly for something as sensitive and complex as online proctoring with fashionable machine learning and artificial intelligence (AI) technologies.

So we had a crack at playing around with the online mock exam, as now also described in some detail in the NLS-NLAT manual (view PDF here).

Fooling the initial facial recognition

Things were off to a great start, when even the Firefox browser and an Ubuntu Linux variant (version 20.04) received checkmarks of compatibility.

NB: Note that the final exam or simulation may not work on such browser and OS though, as the official technical specifications recommend Google Chrome and Ubuntu versions up to 18.04.

Next, it asked us to agree to some terms and allow permissions to our webcam so we can send it a photo of our face.

It turns out holding up a mobile phone with a picture on it worked very well to get past that part, though it was smart enough to detect that a cat face was not a human face.

However, it was perfectly happy accepting a picture on a mobile phone of any humanoid face, including even infamous Marvel-universe supervillain Thanos instead.

Of course, we are being slightly facetious here (even a moderately competent proctor would notice Thanos is not a real candidate) but it’s clear that at least the mock’s mugshot technology is not some advanced piece of cutting-edge AI but fairly off-the-shelf facial recognition.

It then requests the upload of an identity card; in the mock at least it seemed satisfied with any picture you take (though a live proctor might catch any obvious fake IDs).

Fullscreen your browser

Your browser then switches to full screen mode.

This complains and eventually auto-submits your test if you switch to other windows or tabs too often, so make sure you don’t have anti-virus, Windows / software updaters or other software running that might annoyingly pop up windows at random intervals to nag you, and end up ruining your test for you.

On some attempts at the test, in our case and others, the green light on our webcam was still turned on, while on other attempts it turned off after taking a photo of your ID.

But this was only a mock and probably did not include the full proctoring suite, which according to an announcement made by NLSIU today, will be testable tomorrow in a full 'simulation'.

Does the test still have 'magic' software sauce?

In the initial announcement of technical requirements for the exam, back when it was Windows 7 only and before it was rolled out to Mac, Linux and Android operating systems (OS), the technical requirements needed you to download a mysterious software tool. According to the notification:

Extensions: Candidates must download and install Safe Assessment Browser Tool (registered candidates shall be provided a link to download the SAB tool in advance of the examination)

However, when compatibility was extended to other OSes, that paragraph was dropped from the technology requirements, presumably because porting reliable software, which requires administrative system access, for three new OSes on such a short timeline may literally be impossible, even for companies such as a Google.

In remote-proctored exams, such software acts almost like anti-virus software, sitting deep in your system with administrative permissions so that it can monitor everything that is running at your computer, including potentially checking the peripherals connected, recording every single keystroke and possibly checking for screen recording streaming software in the background. We had explained this in our detailed interview about LSAT-India's remote proctored test.

If this software is indeed gone, the NLAT will be at a serious disadvantage in trying to catch even the simplest cheating tools installed on the same computer.

To keep you safe when browsing the internet, which is well-known to be full of nefarious sites trying to steal your personal data and worse, all websites and software running on websites (which includes the NLAT test) accessed via a reputable browser, without any extensions, will be severely limited in terms of the information they can obtain from your system.

A website viewed on a browser is certainly not able to detect even simple things such as screen recording software (and we have confirmed others’ account that it is possible to run screen recording software and screenshotting software on the NLAT mock exam without any problems), nor could it detect whether a second monitor has been hooked up via an HDMI cable (we will get to why these could be an issue further below).

We have reached out to NLS whether the SAB software is still part of the process.

Could such software be exploited?

But even if software such as this were required and NLAT proctoring and exam software was not hamstrung by having to live purely inside your browser, it can never ever be foolproof.

It is a well-known adage of software development that literally no software, no matter how simple, is ever 100% secure and devoid of bugs that can be exploited or worked around.

So, let’s imagine the possibility of a bored IIT student or, even worse, a paid-for professional service, offering software, services or assistance with cracking something as prestigious as the NLSIU entrance test.

How do you stop a skilled IT engineer from coding software that lives on your computer, like a virus, and which secretly captures everything on your screen and sends it to someone else? In fact, such software already exists in near endless variation of off-the-shelf and freely available malware, which cybercriminals routinely use to steal your online banking passwords or to send screenshots or stream your screen to another computer (for instance, to steal your credit card data whenever you are doing online shopping).

If even anti-virus software vendors, which have specialised purely in detecting such malware have a hard time keeping up with such software, can we expect an online testing company doing so.

Saving grace

NLS has a few potential mitigations and advantages, in particular because it only needs to really identify cheaters amongst the top scorers who make it into the merit list of 80 to 120 candidates (depending on whether its batch expansion goes through). This would give it the ability to manually get human post mortem proctors to do very close watches of the video of every single candidate who makes it to the final merit list, in order to spot any anomalies.

We are not sure if NLS is planning this kind of close scrutiny but this kind of oversight would most probably catch any attempt by someone like Thanos to sit the exam.

Update 11 September 2020 14:57: A new section has been added to the FAQs that confirms: “Further investigation post the examination will be carried out with help of audit logs generated on the system, and NLSIU may disqualify the candidate upon such examination”.

In addition, linked with honing in on suspicious high-performers, automatic AI proctoring systems can also work on the basis of detecting candidates’ gaze and where they are looking during the test. That might help in obvious cases, such as typing away on a second laptop with Google on it, but there is no guarantee it will be able to catch a well-practised cheater in the act (particularly with a potentially low webcam resolution of 640x480), nor is it certain that an advanced cheating technique would even have any visual evidence for a camera to catch.

Potentially, performance-linked invigilation could also happen live, with human proctors honing in more closely on candidates who are performing particularly above the average in early sections, asking them to move their webcams around the room, show their desk, etc (a full list of what you are not allowed to do and what proctors can ask you to do, is available in the NLAT's proctoring protocol document (PDF)).

We are not sure if all this will realistically happen, or if invigilators will only step in when the machine learning proctor flags one of the tens of thousands of candidates, but even so, this might end up being unfair for candidates who are simply performing well, and who might get stressed out and lose their groove after such an intervention.

The simple hacks that may still be hard to catch

The real problems start when a deceitful mind starts looking into ways of cheating the system, especially if such minds are also willing to take the risk of disqualification.

Would the systems catch a candidate secretly using a tiny pocket calculator hidden amongst the two permitted A4 worksheets of paper, to gain an edge in the math section? And with maths representing 8 points out of 40 questions in the SLAT (and many lawyers not being terribly good at maths), this could give a serious and tempting edge.

A webcam looking up at a candidates’ face is unlikely to be able to see the desk and papers (and some models of laptops have webcams in such unfortunate positions that getting the camera to even show the desk and a face at the same time may be physically impossible without lifting up the entire laptop or putting your face on top of the desk).

The proctors do have wide powers to intervene and make candidates move the webcam around, according to the proctoring protocol:

1. Proctors will send a message to such candidates: “Please sweep the camera in a circle around the place you are taking the assessment” and / or: “Please tilt the device/ keyboard you are using for the assessment”.

2. Candidates to show a 360-degree view of the entire room by turning on a webcam (laptops/desktops) or front/ selfie camera (as directed by proctor) (mobile devices)/ tilt the device/ keyboard, as instructed.

3. If any unauthorised person or unauthorised device is observed in the candidate’s vicinity, including but not limited to person(s) other than permitted scribes, HDMI cables/ external devices/ projection of the test screen, etc., proctors to terminate the candidate’s examination attempt immediately.

4. If the proctor observes a phone/ tablet/ other device falling/ slipping when the device/ keyboard used for the assessment is tilted, proctor to terminate the exam immediately.

5. Any candidate who fails to comply with the proctor’s instruction within 45 seconds will be logged out of the examination. Proctors may permit 10 additional seconds in situations where they, in their discretion, determine that the candidate may not have noticed messages from the proctor in the course of attempting the examination.

But who’s to say that 55 seconds is not enough for a shifty candidate to apply some sleight of hand and ferret away a tiny calculator?

And with candidates in the several tens of thousands, you might need hundreds if not thousands of human proctors to do an effective job of chasing up candidates behaving dubiously (or just having bad luck with their internet or working in surroundings with insufficient privacy). If you don’t have enough invigilators you will face a real bottleneck in the numbers of real-time interventions your proctors can make.

Update 11 September 2020 14:57: A new section has been added to the NLS FAQs that asks: “What if a student is keeping other devices like a calculator or another phone during the examination? If a candidate is caught looking in other directions continuously, or is suspected to be using or looking at any other device, the candidate will be asked to show the surroundings, tilt the screen or do a 360-degree sweep of the room.”

The Google edge

The advantage that LSAT-India claimed was that none of its questions were supposedly possible to answer by using the help of Google.

Not so in the SLAT, where 8 out of 40 questions are general knowledge (GK) based, similar to the Common Law Admission Test (CLAT), for which candidates have been mugging up for months if not a year or more.

While the initial announcement of the NLAT had claimed that all questions would be “interpretative”, the reality is that the mock sample paper alone, two questions were easily Google-able with two or three keywords from the question, not even requiring a clickthrough.

The temptation of a stressed candidate to whip out their phone and furtively begin typing away to confirm a question they’re stuck on might be too great in some cases, despite the risk of detection.

We have asked NLS for repeated comment on whether the exam will include any non-interpretative questions.

Over-engineering solutions

Besides cheating by oneself with a calculator or Google, the more sophisticated and even harder to detect approaches to cheating on a home-proctored test would involve getting one or more others (who are better than you at exams) to help.

We have been sent the following account from one candidate who shared it with us (and also with NLSIU, asking them to take action). The substance of the email, the veracity of which is fundamentally impossible for us to confirm, was:

1. The alleged (and unconfirmable) plans are that a friend has recruited teachers from local coaching centres and friends to also sit the exam at the same time. So far so good: we can authoritatively confirm, that so far the NLAT registration has allowed pretty much anyone to register without checking documents in detail.

2. “A hidden camera will be used to know which questions will be asked to her (because NLAT has decided to shuffle questions); multiple people will be giving the test at the same time to make sure that at least some of them get the questions asked to her in their respective tests!" the candidate told us (and NLS). “Answers of questions asked to her will be sent via Whatsapp on a phone which will be placed next to screen of the laptop of this girl, so that even if she’s looking at the WhatsApp Messages; AI will not be able to catch it because it’ll think that she’s looking at the laptop screen! They have a very detailed plan and resources to implement it.”

3. ...

4. Profit!!!

Chopsticks FTW

To be honest, the above seems far too complicated and far-fetched and far more stressful cognitively than simpler methods (though all cheating would by nature be quite stressful, if it was actually important to someone to get into NLSIU).

VERY IMPORTANT REMINDER: Please do not do any of this to cheat on any exam ever, as they are all terrible ideas and you will probably be caught. If you are stupid enough to do so, any liability and consequences are your own.

For instance, in order to have access to Google, rubber-banded chopsticks on the primary test-taking phone can hold a second phone, according to possible tech person @kingslyj on Twitter, leaving it permanently invisible to the camera. One could probably also do the chopsticks thing on a laptop. As a complication, one might also need a way to secretly operate the second phone with a Bluetooth keyboard and mouse to actually Google the answers, which might be a tiny bit more difficult, though there are probably dozens of solutions to operating and secreting such objects away when confronted by a proctor, since they will not be able to undertake full cavity searches.

For the more technologically minded, the same tweeter has suggested buying an Arduino, a tiny computer for just over Rs 1,000, which has an integrated camera and can be hidden somewhere near to your laptop to send a live video feed of your screen to someone else (and that’s if you’re not willing to run the risk of just using a screen recorder).

DIY building projects

And there must be hundreds of other more or less technical and/or feasible and/or serious solutions to give someone else access to your screen.

At the end of the day, in most cases, the main technical problem to solve is of effective and subtle two-way real-time communication of questions and correct answers.

VERY IMPORTANT REMINDER: Please do not do any of this to cheat on any exam ever, as they are all terrible ideas and you will probably be caught. If you are stupid enough to do so, any liability and consequences are your own.

• Build your very own so-called Mechanical Turk, via an elaborate system of mirrors and lenses to allow a smart person sitting under (or inside!) your desk to view your screen and tap your leg once to four times, a, b, c or d, for the correct answers.
• In a more modern twist, live stream your screen via the internet with secret screen capturing software directly to your smart friend with laptop sitting under your desk.
• Make friends with ornithologists / astronomers / wildlife photographer and borrow their expensive binoculars / telescope / tele-zoom lens. Have your smart friend sit on the roof or in the apartments opposite and train their lens on your screen through your window. Have them yank a fishing line going from their window to yours, attached to your ear, one to four times for each answer.
• Get your smarter and more studious older brother or sister, who looks a little like you, to sit the test for you. With the resolution of webcam and the quality of most picture IDs, no one might find out unless they choose to do a comparison of the video once you arrive at NLS (which would be very embarrassing).
• Attach a tiny spy camera (or potentially even a standard camera phone) just above or to the side of your laptop screen, pointing down, which can record and stream the content of your screen wirelessly to a computer next door, which can do some video processing to turn the image upside down and make it more legible. Get your local electronics shop to jerry-rig a wireless battery-powered doorbell to just vibrate, instead of ring. Tape the buzzy end to your leg and your friend who actually figures out the answers presses the buzzer as applicable, for each answer on your screen.

Reiterating, the above are primarily far-fetched and unfeasible. And building any of such contraptions or systems is probably a fools’ errand.

But in what can fairly be described as the land of competition, opportunity as well as extraordinary jugaad, the odds are that one in 50,000 may have already built something as ridiculous as the above.

With that kind of risk, NLS will need to be supremely confident that its technology will be robust enough to cope but right now the available evidence suggests otherwise.