Physics GRE Scores of Prize Postdoctoral Fellows in Astronomy

The Graduate Record Exam (GRE) subject test in Physics currently plays a significant role in graduate admissions for U.S. astronomy departments. Out of the 20 top-ranked astronomy graduate programs in the U.S. (National Research Council rankings, 2010; excluding physics departments), 100% of them required the Physics GRE (PGRE) as part of a complete application to their graduate program in 2015. While the role that the PGRE plays in assessment varies among individual departments and admissions committees, applications are often subject to formal or informal “cutoffs” based on the PGRE percentile score (with a perfect score earning a percentile of 99%). This practice is based on the assumption that the PGRE is a useful gauge of whether applicants will be “successful”, both as graduate students and as professionals. Lower-scoring applicants are either eliminated from the pool entirely or required to compensate for a low PGRE score with exceptional credentials in other areas of their application. Prospective applicants with low PGRE scores will sometimes choose to not even apply to particular programs - particularly those which quote an “average” or “typical” PGRE score on their websites - out of a belief that their score will automatically render them uncompetitive in the applicant pool.

Standardized tests similar to the PGRE have a demonstrated record of performance being skewed by gender and race. The Educational Testing Service, which administers the PGRE as well as the General GRE, has demonstrated that both gender and race impact scores on the General GRE, with lower scores among women and underrepresented minorities (see, for example, Steele & Aronson 1995, Miller & Stassun 2014). This same demographic skew has been demonstrated for a broad array of standardized tests, including the PGRE itself¹¹1See http://www.ets.org/Media/Tests/GRE/pdf/gre_0809_factors_2006-07.pdf (e.g. Miller 2013).

The predictive power of such exams for long-term outcomes is also unclear. Quantifying “success” either in or beyond graduate school is a complex undertaking, and reducing this to a single metric is often considered ineffectual. For example, while Glanz (1996) shows that PGRE scores weakly correlate with average coursework grades for physics graduate students at Harvard, the same work also argues against heavily weighting the Physics GRE in graduate admissions procedures for physics programs. Glanz (1996) also demonstrates no correlation between coursework performance and other metrics of “success” in graduate school, such as research productivity.

The role of the PGRE in astronomy graduate admissions - or, more specifically, whether PGRE score correlates with “success” in the field of astronomy - has not been explicitly studied. Demographic trends in the PGRE are hard to examine within small sample groups, and it is difficult to converge upon a satisfactory and broadly inclusive definition of “success”. However, we believe that even an informal examination of this specific scenario - PGRE scores of “successful” astronomers - can be useful in considering how the PGRE should be used in astronomy graduate admissions.

Here we present data from a questionnaire sent to 271 individuals who have held prize postdoctoral fellowships in astronomy between 2010-2015. The questionnaire asked recipients to self-report their scores on the PGRE.

Figure 2 illustrates the difference between the “expected” PGRE scores of prize fellows and those found in our data. On the left, we show the expected appearance of a score histogram for prize fellows if the PGRE was strongly correlated with “success” and a 50th-percentile hard cutoff in graduate admissions was an effective means of selecting potentially successful applicants. In this scenario we would expect scores to be distributed as an approximate half-Gaussian, with most fellows’ scores falling into the highest bins and no prize fellows with PGRE scores below the graduate admissions cutoff score. This estimate also assumes that there is no gender disparity in PGRE performance; while the gender demographics match those of our survey respondents, the score distributions within each gender category are identical.

On the right we show our actual histogram of PGRE percentile scores reported by prize fellows. The score distribution spans the full range of percentiles, with 44 fellows ($\sim$30%) falling below the 50th percentile. The distribution of scores is also notably different for male and female fellows; $\sim$42% of female fellows fall below the 50th percentile. The results show a clear disparity in gender performance on the Physics GRE among prize fellows. Multiple fellowships are represented in each percentile bin.

Figure 3 illustrates the number (left) and fraction (right) of fellows - both in total and divided by gender - that would be eliminated from a theoretical graduate school applicant pool by applying hard thresholds in the PGRE percentiles, with thresholds ranging from 10-60%. Table 1 offers a numerical breakdown of these values, with thresholds ranging from 10-90%.

In addition to fellows’ PGRE scores, we also considered whether PGRE score showed any correlation with first author publications; a statistically robust correlation would suggest that the PGRE may be indicative of research productivity in astronomy. Figure 4 compares the number of first author publications written by fellows at various stages of their career to PGRE scores. While there is marginal evidence ($p=0.047$) of a weak ($r=0.16$) correlation between PGRE score and number of papers published as an undergraduate, there is no statistically significant ($p<0.05$) correlation between PGRE score and total first author papers during or exiting graduate school among 147 prize fellows (2 of the 149 respondents who reported PGRE scores did not report first author paper statistics).

Similarly, Figure 5 compares histograms of first author papers written at various career stages for prize fellows whose PGRE scores fell below thresholds ranging from 20-60%. The mean and median number of first author publications for fellows in each cutoff category are illustrated. The general shape of the histogram and the average number of publications are both insensitive to PGRE cutoff score.

Table 2 offers a numerical breakdown of these results, giving the total number of first author papers written by prize fellows with PGRE scores below thresholds ranging from 10-90%.

Based on the above results, we find no evidence that the PGRE can be used as an effective predictor of “success” either in or beyond graduate school. The PGRE scores of individuals who have held prize fellowships do not adhere to any minimum cutoff, and show a significant deviation from the score distribution that would be expected if PGRE correlated with career performance. In addition, there is also no statistically significant correlation shown between PGRE score and the number of first author publications written by prize fellows either during or exiting graduate school. The average number of first author publications among prize fellows remains unchanged regardless of whether a PGRE cutoff score is applied.

Publication records also cannot simply be used to counteract low PGRE scores in admissions decisions; of the 15 fellows who scored $<$30% on the PGRE, only 3 (20%) had published one or more first author papers during their undergraduate careers. It is also worth noting that this is in agreement with the overall percentage of fellows who reported publishing one or more first author papers as undergraduates (34, or 23%).

It is true, of course, that all the prize fellows included in this study were eventually admitted to a graduate program and successfully completed their PhDs. However, it is important to note that our sample includes prize fellows who received their PhDs overseas or from programs that do not require the PGRE.

While there are many factors at play in assessing a graduate school applicant pool (including hard-to-quantify elements such as CVs, reference letters, and statements of purpose) these data demonstrate that the use of PGRE scores to either eliminate or negatively impact graduate school applications is likely to remove a significant number of potentially successful astronomers from the applicant pool.

For example, we can consider a “hard” cutoff of 60% applied to a hypothetical applicant pool comprised of our poll respondents. Such a cutoff would have excluded 64 future prize fellowship recipients. In addition, while this would exclude 43.6% of the total applicant pool, the cutoff would disproportionately impact women, excluding 60.4% of female applicants. Those prospective applicants would have gone on to write 267 (133) papers during their graduate careers.

Even a relatively low cutoff in PGRE scores would eliminate future prize fellows (4 at $<$10%, 10 at $<$20%). Overall, the data suggest that any application of a PGRE cutoff score in graduate admissions carries with it a substantial risk of eliminating potentially successful future astronomers from the applicant pool.