Sunday, June 24, 2012

Bombshell from IIT-K: Faculty Recruitment through JEE

Thiruvidaimarudhoor Ananthapadmanabhan
An Interesting Times of India exclusive

In an act of defiance, the IIT-K Senate resolved in an emergency meeting this afternoon to use JEE performance as the sole criterion for recruitment and promotion of its faculty members. Experts have dubbed the latest move a high stakes grenade thrown into the low stakes subculture of Indian academia.

Sidebar

The Senate of IIT-K has also recommended to the IIT-Council that JEE should be the sole criterion for selecting IIT Directors as well. After pointing out that the current interview process is totally dysfunctional, the Senate resolution notes that the new method will have a bonus benefit of getting rid of the most hated figure in the IIT system -- the HRD minister -- from the selection process.

* * *

“This decision on faculty recruitment just spotlights what everyone has known all along,” said Prof. Arpan Roy, the Senate’s media advisor. “JEE is the main reason for our reputation worldwide, and we are determined to leverage it to add a shiny new sheen to our faculty ranks.”

“As our alumni, especially the ministerial ones, have said so often, IITs are special because of their undergrad students. Why are they special? JEE, of course!”

There may be more to the IIT-K move, observers have observed. While IIT students tell mild jokes about their teachers, said Prof. Shashibhushan Sahay, an expert on psychology of IITs students at Imsong University, Shillong, they become more outspoken as alumni in expressing their disdain. The Senate resolution, Sahay noted, may well be a cry for respect.

IIT-K media advisor had a more positive spin, though. “The JEE magic will now rub off on our faculty,” said Roy. “When the exam becomes the sole criterion for recruiting IIT faculty, we too will become famous in Silicon Valley."

To another question, Roy responded with a candid admission that he is a great fan of not just JEE alumni, but also Kentucky Fried Chicken, George W. Bush, G.I. Joe, and Rocky IV.

The IIT-K Senate may also have had another motivation in making this move, according to Mr. Narasimhachar Thathachar, a senior analyst at the Institute of IIT ‘Tudies, a Hyderabad based think tank. The new resolution, he says, may well be to meant to impress upon the government that IIT faculty deserve a special treatment not only because they organize the JEE, but also because “they, you know, actually cleared the dreaded exam.”

When asked if a faculty applicant’s PhD and post-doc experience will be given any weight, Roy responded with a firm no. “We don’t give a shit to our UG applicants' past performance, and look how great they are! The same principle must apply to faculty applicants as well. What really counts is how a man performs when the stakes are stacked sky-high.”

To a reporter who pointed out that American universities like Harvard and Yale prefer to use multiple metrics (including whether students’ parents are alumni), Roy responded, “It’s sad to see these universities turn their back on monomaniac Americans.” He praised his Senate colleagues for choosing wisely, and echoed POTUS # 43 in sending a stern message to the other IITs. “You are either with us, or ...," he stumbled before recovering, "... you risk becoming an unknown unknown.”

Roy also informed the reporters that the Senate meeting concluded with a special screening of Independence Day.

At IIT-D, meanwhile, an emergency meeting of the Senate has been called at 11:00 a.m. tomorrow. The notice for the meeting, a copy of which this reporter has seen, states that as an institution founded on British collaboration, it was IIT-D’s duty to follow IIT-K, even if it means invading HRD Minister’s Faridabad farmhouse and yelling to his face, “You want a common exam? Here it is -- common to both students and faculty!”.

These developments have been welcomed by the cram schools in Kota and Hyderabad. Said Mr. Vinod Sharma, a highly celebrated teacher at SuperStar Academy (SSA) at Kota, “We look forward to welcoming aspiring IIT faculty, as well as helping current IIT faculty earn their promotion. We plan to open a branch in each IIT.”

"We will teach them what to aspire to," said Mr. Sharma, who is rumored to earn upwards of half a million rupees a month. He added, however, that SSA will go strictly by merit and admit only those faculty who get high marks in SSAT -- the SuperStar Admission Test.

Sitting next to Mr. Sharma with a wide grin on his face was Mr. Rohit Singh, the owner of Star Academy, also at Kota.

15
Comments:

In another piece of breaking news coming from the Ministry of Human Resources and Development, the ministry has, following an unianimous decision by the IIT Council, decided that percentile ranks in Class 1 board exams will be given 50% weight in all faculty appointments in the IITs. "This will provided the much needed fillip to school education in the country," said minister Sachin Symbol extolling the virtues of the proposal, "while it will also ensure that Coaching Institutes become irrelevant". Professor Siddharth Gautam, a director of one of the IITs concurred, also offering the view that the inclusion of percentiles was based on the unchallengeable assumptions that "aggregate scores are expected to increase" with 'meritoriousness' (it was not immediately clear what this meant) and that "merit distribution was the same in all boards". He also further said that no data was required to validate these assumptions, presumably because they are already encoded in the Standard Model (and also, incidentally, in String theory, just in case).

It was later learnt that in a magical secretarial feat achieved by the Office of the Honorable Minister, the decision of the IIT Council was unanimous and there was no reported "dissent" from the IIT Senates even when more than half of the IIT Senates were almost frothing at the mouth expressing dissent. A patent has meanwhile been filed for this new method of manufacturing unanimous consent, a commodity whose demand seems to be rather high these days.

I am not sure we should just get onto the "IQ" bandwagon. It is not even clear what most IQ test measure, and whether what they measure has more relevance to, say, engineering and science education, than a test of knowledge and application like the JEE.

As for SAT, it is eminently one of the most trainable exams in the world, so I don not quite understand your point about replacing JEE with a SAT like component. Also, I Can say from personal experience that training is almost the only thing that counts for Olympiads (even Mathematics). A look at India's and the USA's record in Olympiads, for example, is instructive. We fare very poorly in Mathematics (where the "sylalbus" and type of problems are quite different from the JEE) while the USA does a lot better with its dedicated Olympiad training programs. China, with an even more dedicated training program does even better. Now in the Physics Olympiad, India does a lot better, almost always better than the USA and closer to China. The Physics Olympiad problems are also closer to what the JEE Physics exams asks students to solve. You can draw your own conclusions. If more proof is required, you might also take into account that the the Physics Olympiad selection camp in India selects a rather high number of students from Kota, and India's medalists at the Physics Olympiad are almost always in the top 100 of the JEE.

So, as I see it, you propose to replace JEE with a couple of tests: one of them almost certainly more trainable then the JEE, and the other with a high degree of correlation with the JEE, and is at least as trainable as the JEE itself.

Thanks, GT Brown for the interesting link. It is actually counter-intuitive. I know the other side of the argument than Harvard, that is the University of Chicago, which chose to go to pure intellectual merits (and basic science which they thought is superior) and even refused to open engineering branches as this was not pure science. 40 years later, the university regrets its decision and now trying to expand its model.

"I am not sure we should just get onto the "IQ" bandwagon. It is not even clear what most IQ test measure, and whether what they measure has more relevance to, say, engineering and science education, than a test of knowledge and application like the JEE."

On the validity of IQ:

http://duende.uoregon.edu/~hsu/talks/ggenomics.pdf

Look up the paper referred to in the pdf:

Data Mining the University, Hsu and Schombert, arXiv:1004.2731

On the validity of IQ as a predictor of academic/intellectual success:

"The lowest score in each category among the 12 theoretical physicists would have been roughly V 160 (!) S 130 M >> 150. (Ranges for all groups are given, but I'm too lazy to reproduce them here.) It is hard to estimate the M scores of the physicists since when Roe tried the test on a few of them they more or less solved every problem modulo some careless mistakes. Note the top raw scores (27 out of 30 problems solved) among the non-physicists (obtained by 2 geneticists and a psychologist), are quite high but short of a full score. The corresponding normed score is 194!

The lowest V scores in the 120-range were only obtained by 2 experimental physicists, all other scientists scored well above this level -- note the mean is 166."

On the correlation between SAT and g (see pdf above):

http://pss.sagepub.com/content/15/6/373

"As for SAT, it is eminently one of the most trainable exams in the world, so I don not quite understand your point about replacing JEE with a SAT like component. "

Untrue. See here:

http://www.ets.org/Media/Research/pdf/RR-80-08.pdf

" China, with an even more dedicated training program does even better."

Ethnic Chinese do pretty well on the IMO even when born/raised in the West. The composition of the US, Australian, and Canadian teams is instructive. I'd wager training can only take you so far.

" one of them almost certainly more trainable then the JEE, and the other with a high degree of correlation with the JEE, and is at least as trainable as the JEE itself."

Disagree. The SAT is a decent predictor of academic success except for very high cognitive threshold fields where students typically max out the SAT.

For the problem of distinguishing between people in the right tail of the intelligence spectrum, the Math Olympiad, I would imagine, is pretty good.

The Roe study you quoted is from the 1960s, and only studied (as far as I can make out) a small set of physicists: Here is a more current account from WIkiepdia:

"The American Psychological Association's report "Intelligence: Knowns and Unknowns" states that wherever it has been studied, children with high scores on tests of intelligence tend to learn more of what is taught in school than their lower-scoring peers. The correlation between IQ scores and grades is about .50."

0.5. That's the correlation you get. Unless there is evidence that JEE has a substantially lower correlation than this (and it probably doesn't), I don't think this is significant at all.

"'As for SAT, it is eminently one of the most trainable exams in the world, so I don not quite understand your point about replacing JEE with a SAT like component. '

Untrue. See here:"

As you yourself pointed out, students working for specialized courses tend to max SAT out. I can safely predict that anybody in the JEE top 10000 or so would comfortably max SAT's quantitative part out. That's what they eventually do to the GRE anyway. So that leaves the verbal part, and it is enromously unclear how useful those are going to be for deciding admission to science-and-technology programs. For all their vaunted reliability, SAT and GRE do not enter the admission calculations of most of the better US universities: Princeton's CS department goes so far as to say explicitly that they only require GRE as part of a university cutoff requirement, and do not use it in admission decisions (mostly because of the "max out" factor, I believe: everybody they would want to take proably already has a full or near full score). At best, a SAT like test can probably serve the same cutoff criteria that board exams currently serve, and its not clear why it would do it better.

'Ethnic Chinese do pretty well on the IMO even when born/raised in the West. The composition of the US, Australian, and Canadian teams is instructive. I'd wager training can only take you so far. '

This is a complete non-sequitor. If you look at the rankings at IMO, it is always the same countries that do the best everytime, and these are also the countries that happen to have the best training programs: Iran, Russian Federation, USA, and China for example. Although it is true that the US team often has a high number of Asians, this is not always true, and the the team does well irrespective of the ethnic composition. Now if your argument is that somehow these countries have inherently more "intelligent" people, then you are welcome to that view. I would contend (partly from personal experience at some Olympiads) that training helps, and helps a lot, as you would expect it to in any kind of endeavour.

"For the problem of distinguishing between people in the right tail of the intelligence spectrum, the Math Olympiad, I would imagine, is pretty good."

Again, on the extreme right tail, maybe yes. But something like the INMO in India (which is by invitation only, and where still only about 10-20% of the total number of participants manage to solve more than 2/6 problems*) would have very limited use as a selection test for university exams.

*I saw the ranklist for my years' INMO. People who had managed 3 problems out of 6 had all qualified. Now anecdotes do not an argument make, but I am pretty sure that my year was no differnt from other years, and this is probably a study someone could easily make.

Also, about the vaunted repeatability of SAT/GRE scores, I would like you to consider the following data point: I was able to improve my GRE scores by about a hundred and fifty points over a period of about a month (from a mock test to the real test). The requisite "increase" in my "IQ" during said month came via memorization of a list of 5000 words that I (and for that matter, nobody in my field) am never going to use in any piece of writing. Or, at the very least, not until the very goal of that piece of writing is to confuse rather than clarify.

I personally felt, though, that I must have lost a few precious IQ points trying to memorize that list :). It was, shall we say, a rather stultifying experience. But then, but for it, I would never have known the crucial difference between "venial" and "venal".

Before coaching industry came up at Kota, Hyderabad, etc. (but Agarwal's at Bombay and Brilliant's etc. elsewhere (Madras?) already existed), in those times, National Talent Search Examination scholars used to be directly admitted to IITs (at least to IIT Bombay). Not just that. The top rankers at the *state* boards, and not just those at CBSE/ICSE boards also used to be directly offered admissions. (Not sure if it was top 10 or 20 0r 30.) Even if the state Boards ranks used to be decided, as always, by performance on all subjects (including English, biology and humanities/social sciences). Apparently, at least one President's Gold Medalist who now has a Wiki page devoted to him, was admitted to an IIT via the Boards route. (No, I don't overestimate either of those two criteria---you should try to get to know me (!!)---but still, things like that do mean something to many people. Including to certain employers. That's why.)

Notice that when educational institutions anywhere in the world have some one main criterion (here, JEE), and still show "generosity" in admitting students using some other criteria, they are always a bit tougher on the *other* criteria (here, NTSE, Board rankers).

The above-mentioned direct admission policies were in effect when the total intake at *all* the (five original) IITs and BHU-IT *put together*, was about 1800, with the engineering departments closing in at about 1200.

Interestingly, those days, typically, one would also read of one/two top 25 AIR still deciding on pursuing an Integrated MSc program, typically in physics or maths, instead of pursuing an engineering program like the herd would (and does even more intently these days), but that's a bit of a digression.

So, let's say: about 1200--1800 for the most sought after programs. And, direct admissions being possible if you were in the top 500 by NTSE, and, even assuming just top 3--4 rankers per board, about 100 by the Boards route. That's about 600 being found OK via the non-JEE route, for a total intake of about 1800. This means, factoring the adverse bias against the secondary criteria out, these routes were actually being regarded as probably just as good as the JEE route.

If the intervening chest-beating of IITs (esp. of the now retired IIT professors) is anything to go by, obviously, these other folks, too, performed very, very well---they were, are, and for eternity will be: "IITians."

Further, BITS Pilani those days would accept purely by normalized Boards marks (all subjects put together, not just PCM). (Normalization by marks or percentiles, I am not sure, but probably it was normalization by marks. I didn't take JEE, would have got BITS Pilani via any way (PCM/total, marks/percentiles), and chose not to even apply. (To JPBTIs: Statistically speaking, there aren't too many who don't even apply, so don't make an issue out of the "measurement" problem.))

You may make an argument that in today's times, going by today's Boards doesn't make sense. Given the general decline of standards in every sphere, one wouldn't need a lot of hard data before accepting that argument for a more serious and closer scrutiny.

However, you can hardly make any convincing argument that:

(i) the IQ-like tests, requiring minimal preparation (with ease of distribution of training material at low cost) and specially designed for high end aren't suited for mere selection (without branch allocation) for the UG program in a country as populous, diverse, and heterogeneous in every sense (economic/cultural/etc. backgrounds included) as India, won't be statistically at least as effective as JEE on all the four elements of the True/False Positives/Negatives matrix;

and,

(ii) in contrast, the extent to which scores on subject-intensive examinations such as JEE can be maxed out/mastered is lesser than that on the Boards or GRE, especially speaking of the JEE of present times.

We focus on the present times, though, having seen more or less not-so-bright but "ghassu" people cracking ranks in top 750 AIR, and moderately bright in top 500 AIR also in our times, one could argue for the JEEs even of late 1970s being not very different from today's JEE.

BTW, speaking of recent times, I had to reject a JEE ~800 AIR rank guy with 2+ years of professional C++ programming experience in India and USA, for not knowing enough of the language that he could be trusted to at least immediately begin maintaining/enhancing the programs I had written, what with his also not being able to crack the simplest logic-testing test designed by a usual kind of a BE other than me. His performance put him in about the mediocre range of folks coming from those numerous "private" i.e. self-financed colleges. And, honest, I didn't keep a bias against him, in fact, looking at his resume and experience, I might have had a positive bias for him to be in, on our team---and, on social front, he was excellent. ... Anecdotes don't necessarily give a large-scale view, but when they offer a qualitative difference as big as that, you have to take a real hard second look.

... Obviously, the past-vs-present argument cuts both ways.

And, BTW, to repeat, never make the mistake of taking the JEE as the standard of evaluating other tests. That's one premise I notice subtly being brought in so many arguments again and again, all over. JPBTIs can be trusted to be idiotic (or crooked) enough not to be able to (or wanting to) detect (or highlight) that part. That doesn't mean the rest of us should give up virtues like honesty or independence---esp. the independence of thought i.e. of thinking.

[I may post this reply as a separate post at my blog, esp. if Abi deletes it---I have saved the contents, Abi.]

(i) the IQ-like tests, requiring minimal preparation (with ease of distribution of training material at low cost) and specially designed for high end aren't suited for mere selection (without branch allocation) in the IIT UG programs, given a country as populous, diverse, and heterogeneous in every sense (economic/cultural/etc. backgrounds included) as India; that they won't be statistically at least as effective as JEE (if not statistically a lot better) on all the four partitions of the True/False-- Positives/Negatives matrix;

Thanks for the reply. I don't intend this to be a rebuttal, but am just spitting out what I think works so that it may be critiqued.

"0.5. That's the correlation you get. Unless there is evidence that JEE has a substantially lower correlation than this (and it probably doesn't), I don't think this is significant at all."

A lot of bunching/ranking based on noise instead of signal makes me somewhat skeptical about the stronger correlation of the JEE than the SAT. This is of course complicated by things like restriction of range by JEE score. You're right though, that an exam that is a function of conscientiousness + smarts should in theory be a better predictor of academic success than one that is predicated mainly (solely?) on smarts (like the SAT).

"As you yourself pointed out, students working for specialized courses tend to max SAT out. "

SAT V+Q seems like a good proxy for general smarts. Only 300 or so students, IIRC, max out the SAT in the US each year. It seems to be a decent job of quantifying smarts for the bulk of students - better than the JEE, based on criticisms that I have read.

"I can safely predict that anybody in the JEE top 10000 or so would comfortably max SAT's quantitative part out. "

Personally I think that the abysmal PISA scores in Tamil Nadu and HP indicate (see Lant Pritchett's article on this or this paper: http://www-wds.worldbank.org/servlet/WDSContentServer/WDSP/IB/2008/06/17/000158349_20080617085945/Rendered/PDF/wps4644.pdf) that this is extremely unlikely to be true. If it were, you're right that the Q section would be useless as a filter. A more likely outcome is that there might be not be much statistical difference in Q scores for many of the students in the top few thousand. I of course still think that V+Q would be a useful metric, as it is in places like Caltech, and the Q difficulty could be cranked up a notch to create separation, but without ranking noise and not signal (the pre-1994 SAT did that). This would probably obviate, operationally, the need for an Olympiad-style examination except that Olympiad exams are a good test for depth of thinking too.

" I would contend (partly from personal experience at some Olympiads) that training helps, and helps a lot, as you would expect it to in any kind of endeavour. "

Preparation is a necessary but not sufficient condition for Olympiad success. The other prerequisite is extremely high intelligence, which this is a good filter for, in order to separate the top few from the rest.

"Again, on the extreme right tail, maybe yes. But something like the INMO in India (which is by invitation only, and where still only about 10-20% of the total number of participants manage to solve more than 2/6 problems*) would have very limited use as a selection test for university exams."

Agreed. Some variant should be able to distinguish between the top few hundred students. The SAT equivalent should work for the rest.

" I was able to improve my GRE scores by about a hundred and fifty points over a period of about a month (from a mock test to the real test). The requisite "increase" in my "IQ" during said month came via memorization of a list of 5000 words that I (and for that matter, nobody in my field) am never going to use in any piece of writing."

I dunno. The data doesn't indicate that such improvements are the norm, but then I'd imagine that someone smart *in the US* would have, by osmosis, known most of the words in the wordlist.