Monday, August 6, 2012

Is neuroscience a meritocracy?

Meritocratic failure

In a recent article, "Why Elites Fail," Christopher Hayes argues that meritocracies often fail. For example, the Hunter College High School, a prestigious public school in NYC, accepts students solely on the basis of an entrance exam. For decades, the meritocratic admissions process meant that the school had a diverse student body: in 1995, 12% of the school was black, and 6% hispanic. Today, however, when rich parents hire private tutors for their kids, the student body has become less diverse, with 3% black, and 1% hispanic.

Hayes identifies two keys to meritocracy. First, there must be interindividual differences in ability, skill, or what-have-you. Second, people must be rewarded for their performance: high performers get promoted, while low performers are "punished." If you looked at how families perform between generations, variance in individual ability and accountability would cause inter-generational mobility: if a parent is exceptional, and their child average, the family would change positions. In theory, the larger the variance in interindividual differences is, the larger the mobility should be (I am not sure how to state this formally and correctly, but that is Hayes argument).

However, this is not what usually happens. After one round of meritocracy, the winners get to invest in the next round, so their children have advantages. Sometimes the winners, often in authority, get to choose the winners (or losers) of the next round. The meritocracy breaks down, and become an oligarchy. This same basic story has played out in many different fields in the last thirty years, including college admissions ("legacy admissions"), or the decrease in inter-generational income mobility in the US since 1970.

After reading the article, I wondered, has the meritocracy has failed in neuroscience as well?

How to test scientific meritocracy

The scientific meritocracy can fail in many ways. Funding can go to prestigious labs, regardless of how efficient they are. Nobel laureates can publish shitty papers in high profile journals. Prizes can go to PIs at "elite" institutions, to reflect how portentous the prize is. (I use "elite" as a shorthand for the top 5-10 neuroscience programs, based on my rankings, and my perception. Exploring what constitutes an elite institution is for another post.)

Since I pretend to be a scientist, I wanted to measure how meritocratic "science" is. One possibility would be to measure whether labs from elite institutions get preferential treatment by journals, but that would require somehow objectively measuring paper quality, and controlling for funding. Instead, I settled on something simpler, and hopefully more objective: looking at the credentials of PIs at elite institutions.

In a meritocracy, you would expect that as you looked farther into the past, the winners would have increasingly diverse backgrounds. Or, looking forward, you would expect some percentage of people who start at the bottom of the hierarchy, but were talented, could work their way up. In terms of PIs at elite institutions, my expectation is that they would almost exclusively have done post-docs at other elite institutions. In contrast, if you looked at where they went to undergrad, I would expect a more diverse set of schools. To see whether this was true, I looked at 30+ PIs at elite institutions.

The test

The baseball writer Rob Neyer has a gimmick where he presents stats for two anonymous players, "Player A", and "Player B." Sometimes the stats would be similar, and the reader would be shocked to find that Player A was an All-Star, while Player B was a "scrub." Sometimes the stats would be dissimilar, but Players A and B would actually be the same player, playing under different conditions. The point of the gimmick was to look at the world more objectively, without the halo effect of their names.

So in that vein, I present two cohorts of researchers, and their associated institutions:

Cohort A Cohort B
MIT
Santa Barbara
Stetson
Cal State Chico
Williams College
Cambridge
Lawrence
Vassar
Duke
MIT
Harvard
MIT
UChicago
Stanford
Bryn Mawr
Yale
Harvard
CalTech
UChicago
Vanderbilt
Harvard
Brown
Princeton
Berkeley
Berkeley
UVA
MIT
UVA
Harvard


With all due respect to the universities represented in Cohort A, most people would agree that the schools in Cohort B produce more research. So who are these two cohorts of researchers, and how are they affiliated with the institutions? Both cohorts are professors at elite neuroscience departments. But Cohort A got their bachelors before 1990 while Cohort B got theirs after 1990:

Fogies Undergrad Whippersnappers Undergrad
Barres, Ben
Knudsen
Newsome, William
Moore, Tirin
Raymond, Jennifer
Shatz, Carla
Nicoll, Roger
Huganir, Richard
Bear, Mark
Julius, David
Malenka, Rob
Tsien, Richard
Katz, LC
Callaway, Ed
Cline, Hollis
MIT
Santa Barbara
Stetson
Cal State Chico
Williams College
Cambridge
Lawrence
Vassar
Duke
MIT
Harvard
MIT
UChicago
Stanford
Bryn Mawr
Datta, Bob
Wilson, Rachel
Ehlers, Mike
Scott, Kristin
Harvey, Christopher
Deisseroth
Dolmetsch, Ricardo
Heiman, Miriam
Huberman, Andrew
Potter, Chris
Shuler, Marshall
Tye, Kay
Goosens, Kim
Sabatini, Bernardo
Yale
Harvard
CalTech
UChicago
Vanderbilt
Harvard
Brown
Princeton
Berkeley
Berkeley
UVA
MIT
UVA
Harvard

(Methods: To select fogie professors, I included professors I recognized by name, or who are HHMI investigators. For the whippersnappers, I used the SFN Young Investigators award listing, and scanned the websites of departments for assistant professors. People educated outside the US were excluded. For each professor, I noted their schools for BS, PhD, and post-doc. For some professors, I could not ascertain their undergrad institution, and excluded them. This is by no means exhaustive, but I only spent a few hours doing this. Full spreadsheet.)

I have two general conclusions from this. First, if you want to be a professor at an elite institution today, you need to have gone to an elite undergrad. The "worst" school represented, UVA, is ranked #25 (for whatever rankings are worth). There are no Wisconsin-Madisons on the list, let alone places like Ohio State. Three steps removed from your PI position, where you did your undergrad is a determining factor for whether you can become a PI. As you move the credential window forward to grad school and post-docs, the credential threshold gets even higher.

Second, and more weakly, I think this shows that science has become less meritocratic over time. At first blush, I thought the fogies' schools were just generally worse. A quick Google, however, revealed that Williams, Vassar, and Bryn Mawr are well regarded liberal arts schools. So while it's not fair to conclude that the older cohort went to worse schools, I think it is fair to say they went to a more diverse set of schools, ones that did not necessarily emphasize research.

Questions I ask myself

Isn't this sample size small? Yes, but then again there aren't many professors. If I wanted to spend more time, I should probably quantify rankings of both the PIs' institutions, and their undergrad schools. It would also be helpful to look at non-elite institutions' PIs to see how what is happening there.

Don't elite undergrads reflect high SAT scores/intelligence? And science is a g-loaded occupation, so... I would argue that neuroscience is not as g-loaded as other fields like physics or computer science. Once you reach +1 or +2 SD, other factors like work ethic become important. A quick Google shows that elite institutions have, a ~100 point SAT score gap over other places, ~1SD. However, there are much fewer elite institutions, and they also have fewer students, so by numbers alone, there should be just as many equally smart kids at non-elite institutions as the elite ones, and a much larger group at -1SD.

Ok, if the elite kids aren't smarter, could they have some other trait? I can see this. College admissions are increasingly (and insanely) competitive. Thus, elite colleges may screen for competitiveness. If not competitiveness, it could be some other factor. Elite institutions use extracurriculars as differentiators between applicants, and if you believe this Gladwell piece* the extracurriculars reflect something real.

* What happened to Gladwell? People love to shit on him now, but I thought he did a good job summarizing social science for a wide audience in The Tipping Point and his older magazine articles. But now he's publishing half-baked essays on "slack," and name-checking Tyler Cowen's econ-foodie book?

What about non-Americans? I don't know much about this. My understanding is that the European college system is much more equal in terms of quality, so there is less fighting for spots (with exceptions like the French/Swiss ├ęcoles, Oxbridge,  etc.). As for Asia, I think a majority of Chinese PIs in the US come from Tsinghua or Peking University, and the Indians from IITs. But American grad schools may not be equipped to identify good applicants from less famous schools.

Isn't this a lot of words for what amounts to glorified googling? Yeah.

Concluding bloviating

As mentioned at the top, society as a whole is becoming less meritocratic. It would be remarkable for science to resist this trend. I'm not sure what, if anything can be done about it. The PIs at elite institutions are generally smart, motivated people, so from the perspective of funding agencies, why should they care whether the PIs have diverse backgrounds? And the NIH does fund non-elite institutions, just less so, if only to avoid senators asking why Idaho doesn't get any funding.

There is an opportunity for disruption here, in that elite institutions are completely overlooking talented, less credentialed people. Some places, like Washington University, seem to specialize in being less famous, but nearly as productive, and I think it's in large part by finding people the elite institutes can't be bothered with. Of course, they will still lose status contests to elite institutions in publishing and prizes.

On a personal note, I knew coming to Geneva that I would need to do a second post-doc to get a job back in the US. Seeing the credentials of these people made me realize just how important status and political connections are, rather than simple productivity. Hopefully, the status requirement are much lower one step down the ladder. The post-docs I knew at Duke were able to get positions at good schools like UNC, Baylor, and BU. Whether they would have been able to get the positions if they were post-docs at those schools is another question.

3 comments:

  1. I kind of remember reading a review of the Why Elites Fail Book. Wasn't the high school like 50% asian though? And weren't a lot of them children of immigrants?

    I wonder if some of this may be due to the increasing popularity of neuroscience? I don't know enough about the history of our field, but perhaps pre-1990 there were more jobs and less competition, so of course you're going to get a wide spread of hires - who eventually show themselves to be "better" than their school (so to speak).

    Simultaneously, perhaps the distribution of talent at the other schools was more uniform; were the bidding wars for top faculty as fierce as they are now? I don't know.

    Also if you think this is bad, you should see what it's like in Economics. They seem to be unrepentant in only hiring people from top 5 schools.

    ReplyDelete
  2. I think funding may explain the crunch more than popularity (and much of "neuroscience"'s popularity is really fMRI/cognitive science). The NIH increased science funding pre-Bush, leading to a wave of PhDs, but the PI positions never materialized.

    Also, any popularity effects would probably be transient. Computer science was popular during the dot-com boom, then promptly fell off a cliff in the 2000s.

    ReplyDelete
  3. Interesting analysis - really happy to see this "glorified googling" :)
    i have been formulating this idea for a while that it seems, in science, the quality of institution where you get a PI position seems closely attached to where you did you PhD - no matter what kind of post doc - not even if it was a super-duper Cell/Nature/Science output post doc in a nobel prize/HHMI/BSD lab.
    A little depressing to see that its goes back even further to your undergrad.

    ReplyDelete