The Philosophers' Cocoon

A safe and supportive forum for philosophers.

Owned & Moderated by Marcus Arvan (University of Tampa)

Contact: marvan@ut.edu

Blog mission & moderation policy

IMPORTANT NOTE on how to ensure anonymity in commenting
Job-Market Reporting Thread (2025-26)
Job-Market Discussion Thread (2026-27)

recent posts

about

What’s Wrong with Rankings?: 1. A Measuring Cup Is a Poor Ruler

By Sam Duncan

I believe rankings in general are harmful to higher education and that potential graduate students in philosophy are making a huge mistake if they rely primarily on the Philosophical Gourmet Report to choose a program. But I’ve been thinking about a challenge posed by Shen-yi Liao’s comment on a post about ranking obsession that ran a while back and I think it deserves a response, which is what I will do in this and a few succeeding ones. Liao points out that:

If you make enough comparative judgments, and some of those comparative judgments are enough shared amongst enough people, what is that but a ranking, even if it is not explicitly numericalized? Perhaps there are really people out there who never make any comparative judgment, but I would be surprised.

He raises an excellent point and as I’ve thought about it I’d push it further. We all make comparative judgments– we simply couldn’t get through life if we didn’t– and most of us fall back on rankings and reviews in making some of them. As warmer weather rolled around last year I decided I wanted a cold brew coffee maker. What did I do? I googled “best cold brew coffee maker” and checked the rankings that popped up. I recently wanted to buy myself a really nice ink pen as a 45th birthday present to myself, so what did I do? I checked some reviews and rankings. When my wife and I needed a bigger car to schlep our twins what did we do? We checked the Consumer Reports car rankings.

In none of these cases do I regret using rankings. The cold brew maker is solid and I love the pen. My wife’s reasonably happy with the car, even if she wishes we had the money for a minivan. What’s different about turning to a ranking of philosophy programs like PGR or even whole colleges like U.S. News or its host of imitators? Quite a lot actually.

I plan to get more properly philosophical in later posts but my first reason for thinking that the PGR and U.S. News are flawed and potential students shouldn’t rely on them is simple and compelling: Neither measures what prospective students are or should be interested in. I will focus on the PGR since it is both of more specific interest to this blog and there are already a number of excellent critiques of the U.S. News rankings. I would go so far as to say no potential graduate student should rely primarily on the PGR in choosing a program and to see why let me switch gears for a second and, like a good philosopher, present a slightly outlandish hypothetical. Suppose I told you that I’m very concerned about my blood pressure and so I’ve decided to weigh myself every morning. What would you say? Almost certainly that I should buy a blood pressure cuff and take my blood pressure instead. What if I responded that I had a blood pressure cuff but that I didn’t think it was very good and I worried about its accuracy while my scale was top of the line? Suppose I also pointed you towards a few papers arguing for correlation between blood pressure and weight? No doubt, you’d still think the way I was addressing my concerns absurd. Unless the blood pressure cuff was so bad that the numbers it spat out bore little to no relation to my actual blood pressure it’s obvious that I should use it and not the scale. And if the cuff were that bad I should get a new cuff rather than relying on the scale.

This may seem silly but it’s exactly what relying on the PGR to choose a graduate program is like. The first concern of most potential graduate students is whether or not they will get a tenure track, or at least well-paying and stable, academic job. But of course that’s not what the PGR measures. Instead, it measures the perceived prestige of the programs. It’s a scale. On the other hand, the report APDA (Academic Philosophy Data and Analysis), which has been around for about six years now, directly measures permanent placement. It’s a blood pressure cuff.

This very simple fact is why a lot of the debates that spring up around the PGR are fundamentally silly and not worth engaging with. Is the way APDA measures placement imperfect in various ways? Well even if that’s true the solution is to get better at measuring placement not to fall back on measuring prestige. Is prestige correlated with placement? Well almost certainly, but why rely on something correlated with the thing we’re interested in when we can measure the thing itself?

I can think of two counters to this that actually deserve to be taken seriously. The first, is the fact that placement data reflects factors that may well have changed, or will change, between the current crop of graduates and when this year’s crop of potential graduate students are on the market. This is true, and it’s one reason that even placement data should be treated critically, but it’s hardly a defense of using the PGR or other prestige based rankings. After all, the things that are reflected in the PGR can change just as much over time. What’s worse, the PGR is updated on average less frequently than the APDA (though unfortunately it is also getting a little long in the tooth). The other is that placement is not the same thing as career prospects which is a much harder thing to define or pin down. This is quite true, but rather than a defense of rankings like the PGR it’s a reason why no one should rely on rankings even if those rankings were better than the PGR. I’ll explore this point in my next post.

Posted in ,

20 responses to “What’s Wrong with Rankings?: 1. A Measuring Cup Is a Poor Ruler”

  1. Recent applicant

    “Is prestige correlated with placement? Well almost certainly, but why rely on something correlated with the thing we’re interested in when we can measure the thing itself?”
    I think the argument in favour of this is that past placement performance is not the thing itself. In the terms of the analogy, I’m worried about my blood pressure, but APDA only measures the blood pressure of other people like me. What we’re interested in is what our PhD will mean on the job market 5-6 years from now. There’s no way to measure that through the thing itself; we need a proxy that projects into the future. The question becomes whether APDA or the PGR is a better scale, because neither one directly measures the thing we care about, like a blood pressure cuff.
    There is an argument that PGR is a better scale. If peer assessment of quality of faculty really is correlated with placement as you concede, then, since the placement record of a department will change over time as the faculty changes, current and future-projected quality of faculty might be thought to be even more valuable of a metric than past placement.
    Of course, it would have to be shown that peer assessment of quality of faculty is in fact a better projector than past placement——and I doubt that this would be shown. When I made my decision this year, I placed more weight on APDA’s past placement as a projector, so I’m sympathetic with the general ethos of this article. But I still think that’s an empirical question about which scale is better, not a blood pressure cuff.

    Reply
  2. spell

    It’s Shen-yi, not Shin-yi.

    Reply
  3. Richard Y Chappell

    I mostly see the PGR as an improvement upon asking your local faculty advisor, “Which grad programs have the best philosophy faculty (overall / by subfield)?” which seems like a reasonable question for an applicant to take into consideration.
    (I don’t think anyone recommends considering nothing but PGR ranking.)

    Reply
  4. Grad student

    Here’s a point I haven’t heard considered in this debate before. I suspect that rankings like the PGR make American philosophy students (and then later professors, in the case of those students who are successful in this way) more insular and less knowledgeable as a group (in various respects) by discouraging them from studying abroad. Even if there’s a great program abroad that is the best place for an American student, there is often a fear (probably a fair one) that without that program being named in the PGR their post-graduation prospects will be harmed in the US. This suspicion is regularly confirmed when I talk to American students about grad programs.

    Reply
  5. David Thorstad

    I’m with Richard on this one. Those of us who didn’t go to R1s for undergraduate study didn’t have much of a clue what the top programs were. The PGR gave us a moderately-accurate sense of what those programs were, including some surprising programs (NYU, Rutgers, Pitt, USC) that we wouldn’t have known about.
    It’s certainly possible to over-rely on the PGR. But without the PGR, many undergraduates would be quite badly mis-informed about the lay of the land and would apply to all of the wrong places.

    Reply
  6. Sam Duncan

    Recent applicant,
    I’m not sure I understand the objection here. The Gourmet doesn’t measure future projected quality of a department, but only it’s current perceived quality. Why think that projects into the future? The perceived quality can change just as radically as placement prospects if not more so. I mean no one thinks that Australian Catholic University has the same perceived quality now that it’s closed the Dianoia institute as it’s listed as having in the PGR do they? I knew faculty who told me tales of at least two or three other departments whose prestige dropped just as radically when say a supportive university president retired or even just a coterie of friends left en masse. For example, I’m told that prestige-wise UIllinois Chicago went from being a Rutgers level powerhouse to just respectable in a fairly short space of time, though this happened pre-PGR so no one quantified it.
    Richard,
    I think the subfield lists in the PGR might be the only useful thing about it. I’ll admit that even for areas I know something about those lists might reflect consensus much better than my own judgments or yours or any one person’s for that matter. For instance, I’m guessing you and I would probably rate departments with a lot of utilitarians on the faculty a bit differently in ethics and related fields. They’re probably more complete too. I’ve published a bit on Hegel and a lot on Kant, but good lord I wouldn’t for a second pretend to be able to give you a complete list of departments that are good or even decent places to study Hegel or Kant. But that’s not even the main thing the PGR claims to do. The main thing it claims to do is the rank departments full stop. If it junked the pretensions to ranking departments’ overall quality and just kept the specialty lists then I think that would be a useful thing.
    Although to be truthful, much less useful, it’d need to break those specialties down a lot more finely than the current one does. For instance, it’s just ludicrous to roll bioethics, business ethics, environmental ethics, and AI ethics, and several other subspecialties into one giant ball called “Applied Ethics.” I’d confidently list bioethics as an AOC. I know very little about environmental and business ethics though.
    And maybe no one thinks that students should use only the PGR, but I’ve met plenty of folks who think it should be the main thing one uses.

    Reply
  7. not a blood pressure cuff

    Isn’t the PGR or other ranking list more analogous to a study, however reliable that says whether a school will cause high blood pressure rather than a blood pressure cuff?

    Reply
  8. Cal

    I agree with Richard and David. The best argument for the PGR is to ask how things would be without it.
    Also, in the absence of the PGR, prestige would still influence applications. But only because uninformed undergraduates would have to use global cachet as a proxy for quality of the program. So places like Harvard and Oxford would still be considered by everyone, but places like NYU and Rutgers, whose global prestige diverges from their PGR prestige, would not be on a lot of applicants’ radars.

    Reply
  9. Bill Vanderburgh

    When I was in faculty development, I sometimes worried that the methods for deciding who wins teaching awards were completely inadequate: Riddled with bias (number of student nominations, or even student eval scores) and unable to measure relative quality (there was no mechanism for even looking at the actual teaching of the candidates, let alone coming up with rational criteria for comparing teaching across courses, disciplines, levels, etc.). I eventually made peace with it by realizing that “Teacher of the Year” doesn’t mean the best teacher of the year, but the deserving teacher who happened to get recognized this year. Everyone who won (nearly everyone who was nominated) was an excellent teacher, even if there was no way to say they were really the best, and even if the selection process was deeply flawed.
    I kind of think of PGR like that. No one really knows all those departments well enough to reliably judge that A is better than B in every possible pairwise comparison. There are all kinds of biases, from accidents of acquaintance to anchoring effects to the size of departments to who gets asked to rank and so on. That said, probably, any department that makes the list is good. (I wouldn’t infer, though, that a department not on the list is not good.)
    So, I wouldn’t encourage undergrads NOT to use PGR, just to not take it very seriously. The actual differences between a 7th- or 25th- or 40th-ranked department are small and difficult to nail down. For subdiscipline rankings, I’d treat them merely as indications that you can take a few courses and get supervision in that area if it is one you are interested in.
    Perhaps most importantly, I would want prospective grad students to know that department reputation is a pretty small part of what matters in having a good grad school experience. Finding fellow students you gel with, there being professors willing to supervise you productively, getting opportunities to teach your own courses, being in a city you enjoy, getting full funding, completion rates, average time to completion, job placement rates, and lots more, all strike me as more important than the relative reputational ranking of the departments you are considering as measured almost a decade before you’ll graduate.

    Reply
  10. Recent applicant

    As I understand it, the PGR does project in a small way. If someone’s going into phased retirement and so won’t be advising, assessors are informed of that ahead of time. That’s something that placement record can’t directly capture, because the recent 2023 grads of Superstar but Retiring Professor X can be expected to do very well on the job market but that information could be downright misleading for you as a 2024 applicant. Ditto with lateral moves. And that’s something that quality of faculty rankings updated with that retirement or lateral move information can at least in principle warn 2024 applicants about.
    (I take the point that the PGR isn’t updated that frequently, but that’s contingent, not a critique of the metric itself. Someone could use the above considerations to argue that the PGR should just be updated every year.)
    In any case, I didn’t mean to rest much on the future aspect, since neither metric can project far enough into the future. The key point of the objection is just that neither metric directly tells you what programs will be most successful on the job market 5-6 years from now, and so neither one is best analogized to a blood pressure cuff, i.e. something that directly measures what you care about.
    So to focus just on the metric itself, let’s assume a world where both the PGR and APDA are perfectly up to date every year. The thing that needs to be proven is that past and very recent placement performance would, in those circumstances, be a better projector of job market success than peer assessment of quality of present and very near future faculty. And I guess I’m suggesting that the mere fact that one is a placement record and one is not a placement record shouldn’t be taken to prove that, for the reasons mentioned above.

    Reply
  11. Craig Agule

    Sam,
    I think the key objection here that others are raising is to the central move in the argument, that “my first reason for thinking that the PGR … [is] flawed and potential students shouldn’t rely on [it] is simple and compelling: [It does not] measure what prospective students are or should be interested in.” And your argument is that it does not measure what students are or should be interested in because it measures one thing (reputational quality of faculty) whereas students are or should be interested in another thing (an institution’s placement potential for them).
    But that cannot be a convincing basic and simple argument, because that argument applies in that form equally to every alternative you might offer. The APDA measures one thing (descriptive statistics as to historical placement) whereas students are or should be interested in another (placement potential). Likewise for conversations with faculty at visit weekends, for advice from advisors, etc., etc. Every measure that is on offer is, at best, an indirect measure of what, according to you, students are or should be interested in. Insofar as the argument is the basic and simple argument, it leaves students with literally nothing they can use! That can’t be right.
    In your response to comments, it turns out that your argument is neither basic nor simple. It turns out that, given that the APDA and every other alternative or complement to the PGR is also subject to the basic and simple version of the objection, the real devil has to be in the details. Which measure is the better indirect measure (including, of course, compound measures, which I assume for many include the PGR as but one component)? And that argument is far from basic or simple.
    So, my sense is that you are getting two elements of pushback, one to the presentation of your argument, that it is a bit of false advertising, and another as to the substance of your claim that past performance is clearly better correlated with future potential than with reputational quality.
    None of this is to say that the APDA is not a great tool or that potential students should rely on the PGR without considering other data. Consider everything! But there isn’t going to be a simple, clean objection to any of the not wholly implausible correlated indirect measures.

    Reply
  12. Mahmoud Jalloh

    I agree with a lot of the pushback. I think you plan to get to this with the “career prospects” bit, but a common mistake in attempted dunks on the PGR is that they treat all jobs as the same or make claims that “prestige” is a fake property with no relation to anything else. Students prefer (this preference is obvious due to the differential competitiveness of the market) “prestigious” jobs because they are better (on average). Most people would chose a 2-1 job at MIT over a 4-4 at directional State U for good reasons. Prestige correlates with: lower teaching loads, more pay (and fringe benefits), better cognate departments, more desirable cities, better students, and yes more social recognition.
    Also the fact that the APDA agrees pretty well with the PGR and even more so when placements are limited to R1 or PhD granting institutions (IIRC), which are imperfect proxies for “prestige”, indicates the the PGR is an ok measure of something. (I imagine this improvement in correlation would also appear if placement results were limited to prestigious SLACs.) The default hypothesis is that something like “philosophical quality” is real, that we can recognize it, and that hiring committees can recognize it. Individual and collective judgments may be imperfect, but I don’t think people should be so skeptical of the simple explanation.

    Reply
  13. Sam Duncan

    Bill Vanderburgh,
    I think that’s a pretty good way of looking at this, but I’m not sure I’d grant that any program that makes the list must be good. If I were applying for graduate school right now I’d want at the very least to have better than even odds of getting a job. I checked and at least 9* schools on the PGR list that have ten year placements that are 50% or under.
    I’d also say that in going through the APDA I started to notice what looked like patterns that might be useful for potential graduate students to know. For one, pretty much all UK schools seem to underperform quite a bit from what one would expect given their reputation and/or faculty. You wouldn’t pick that up from the PGR, but it’s something very important for any potential graduate student to be aware of.
    *The nine I saw that were in the top 50 that had 50% or worse ten year placement rates were:
    Brown (43%)
    CUNY (46%)
    UC Davis (38%)
    UC Santa Barbara (34%)
    UIllinois Chicago (38%)
    UC Irvine (26%)
    University of Maryland College Park (26%)
    University of Miami (50%)
    UChicago (50%)

    Reply
  14. can we forget about rankings?

    I started a ranking thread a while ago and expressed strong sentiment against ranking. I do not think PGR is unhelpful or useless. What I was/am concerned about is the obsession with ranking that hurts the community and needs to be addressed, especially among prospective grad students, and also conference attenders. As far as I know, the obsession has not changed in the past ten years. Maybe it cannot be solved by abandoning PGR, but it needs to be addressed by some other means.
    P.S. Maybe it is me myself who is obsessed and projects this onto others? This would explain why my perception has stayed the same over the past ten years.

    Reply
  15. what’s the beef?

    I don’t get your beef with the PGR. You say: “You wouldn’t pick that up from the PGR, but it’s something very important for any potential graduate student to be aware of.” Sure, but Leiter (nor anyone else) has ever suggested the PRG be your sole source of info when it comes to picking a program. As RYC says: unless you think prospective students shouldn’t ask, e.g., their random graduate advisor for advice on programs, I don’t see how you can sensibly begrudge the PGR which just does that in a systematic way.
    The APDA data is also, it seems to me, pretty useless. It just provides lists of placements without giving the context of those who weren’t placed. (No wonder Oxford comes out on top given how enormous a Phd program it is!)
    (The talk about placement vs PGR is also a bit strange in general when the fact that most ever program provides its placement data is largely down to Leiter’s efforts decades ago.)

    Reply
  16. Sam Duncan

    what’s the beef,
    You say:
    The APDA data is also, it seems to me, pretty useless. It just provides lists of placements without giving the context of those who weren’t placed. (No wonder Oxford comes out on top given how enormous a Phd program it is!)
    I’m not sure what you mean by context. Do you mean that we don’t know what happens to graduates who don’t get jobs? If so, I very much agree that schools need to do better at telling us what sorts of non-academic jobs their graduates who aren’t placed in permanent academic jobs get. A lot of programs still fudge on this by not listing them, burying them at the end, or, most commonly, listing no placement for them except the VAP or postdoc they got when they graduated. This is something they need to change and I’m all for putting pressure on them to be more forthcoming about this. But the APDA does tell us what percentage of graduates don’t get academic jobs so it tells us something here, and something the PGR doesn’t. More importantly, is the PGR going to tell us anything about what happens to those who don’t get academic jobs? How exactly is it supposed to be better?
    Or do you mean something else by context? If so, then what exactly do you have in mind?
    Also, why do you say that “Oxford comes out on top”? When I looked just now they have a 47% placement rate. That’s not on top by a long mile. In fact, it’s pretty much the average for programs (46%) and signifcantly worse than some programs that don’t even make the PGR list like say Baylor (71%) the University of Tennessee (67%), and the University of Kansas (67%). In fact, Oxford’s surprisingly mediocre placement is one of the things that makes me think that there’s some systematic thing going on that’s causing UK universities to have trouble placing graduates. Which number from the APDA are you using when you make that claim?

    Reply
  17. UK v. US

    @Sam Duncan re: UK:
    I don’t have numbers but here are some conjectures:
    1. UK phd students might not apply as widely as US phd students. They might be focused more on their non-North American home countries. This is a smaller market, and so it’s harder to succeed.
    2. It’s hard to compare US phd students to UK phd students for at least two reasons:
    A. US phd students usually have a lot more time, and can use that to boost their success rates. It’s not an exaggeration to say that of those who get a permanent position, it’s better to compare US phd students to UK phd students who also did a postdoc afterwards. Maybe related, I think it’s rare for UK assistant professors (lecturers) to be ABD whereas it’s common in the US; people who break into the UK permanent market tend to be already well into their careers.
    B. UK phd students and even postdocs typically handle nothing close to what US phd students (and many postdocs) handle in terms of teaching. That difference in teaching experience obviously bears on success rates for most jobs, where teaching rather than research is primary.

    Reply
  18. academic migrant

    Leiter has a recent post asking about “Hiring of non-US PhDs in US departments”
    https://leiterreports.typepad.com/blog/2023/12/hiring-of-non-us-phds-in-us-departments.html
    Daily Nous has an older post “Placement Patterns in the UK Philosophy Job Market”
    https://dailynous.com/2019/02/14/placement-patterns-uk-philosophy-job-market/
    Both seems to be relevant to this discussion.

    Reply
  19. Sam Duncan

    UK vs. US,
    Those all sound like reasonable hypotheses to me. One thing I plan to post on eventually (if my kids will stay healthy and in daycare long enough for me to write) is how important teaching experience is. I’d also conjecture that there is something of a prejudice against foreign PhD’s in the U.S., though of course that wouldn’t explain why Cambridge and Oxford are having a bad time since I doubt it applies to them.
    I’d also wager that Tory austerity plays some role. Higher ed funding in the U.S. isn’t where I’d like it to be but most states here have upped funding for higher ed since the Great Recession’s huge cuts. The Tories have cut. I’ve also heard that the fall in foreign students in the UK after Brexit has created some economic woes for British universities.

    Reply
  20. Richard Y Chappell

    re: “overall rankings”, one interesting practical upshot of them is that they help with co-ordination. Even if they weren’t initially measuring anything real (which seems an overstatement to me), the mere tendency of students to tend to favor higher-ranked departments means that applicants can expect to be surrounded by better grad student peers if they go to a higher-ranked department.
    (Egalitarians might not like this causal upshot of the rankings, and prefer applicants to be distributed randomly. But I’m generally in favor of “streaming” in education, including in this instance, since I expect it results in higher peaks. YMMV!)

    Reply

Leave a Reply

Discover more from The Philosophers' Cocoon

Subscribe now to keep reading and get access to the full archive.

Continue reading