in academics 1,166 Words

A few thoughts on CSRankings.org

(Warning: nerdy inside-baseball academic blog post follows. If you’re looking for exciting crypto blogging, try back in a couple of days.)

If there’s one thing that academic computer scientists love (or love to hate), it’s comparing themselves to other academics. We don’t do what we do for the big money, after all. We do it — in large part — because we’re curious and want to do good science. (Also there’s sometimes free food.) But then there’s a problem: who’s going to tell is if we’re doing good science?

To a scientist, the solution seems obvious. We just need metrics. And boy, do we get them. Modern scientists can visit Google Scholar to get all sorts of information about their citation count, neatly summarized with an “H-index” or an “i10-index”. These metrics aren’t great, but they’re a good way to pass an afternoon filled with self-doubt, if that’s your sort of thing.

But what if we want to do something more? What if we want to compare institutions as well as individual authors? And even better, what if we could break those institutions down into individual subfields? You could do this painfully on Google Scholar, perhaps. Or you could put your faith in the abominable and apparently wholly made-up U.S. News rankings, as many academics (unfortunately) do.

Alternatively, you could actually collect some data about what scientists are publishing, and work with that.

This is the approach of a new site called “Computer Science Rankings”. As best I can tell, CSRankings is largely an individual project, and doesn’t have the cachet (yet) of U.S. News. At the same time, it provides researchers and administrators with something they love: another way to compare themselves, and to compare different institutions. Moreover, it does so with real data (rather than the Ouija board and blindfold that U.S. News uses). I can’t see it failing to catch on.

And that worries me, because the approach of CSRankings seems a bit arbitrary. And I’m worried about what sort of things it might cause us to do.

You see, people in our field take rankings very seriously. I know folks who have moved their families to the other side of the country over a two-point ranking difference in the U.S. News rankings — despite the fact that we all agree those are absurd. And this is before we consider the real impact on salaries, promotions, and awards of rankings (individual and institutional). People optimize their careers and publications to maximize these stats, not because they’re bad people, but because they’re (mostly) rational and that’s what rankings inspire rational people do.

To me this means we should think very carefully about what our rankings actually say.

Which brings me to the meat of my concerns with CSRankings. At a glance, the site is beautifully designed. It allows you to look at dozens of institutions, broken down by CS subfield. Within those subfields it ranks institutions by a simple metric: adjusted publication counts in top conferences by individual authors.

The calculation isn’t complicated. If you wrote a paper by yourself and had it published in one of the designated top conferences in your field, you’d get a single point. If you wrote a paper with a co-author, then you’d each get half a point. If you wrote a paper that doesn’t appear in a top conference, you get zero points. Your institution gets the sum-total of all the points its researchers receive.

If you believe that people are rational actors optimize for rankings, you might start to see the problem.

First off, what CSRankings is telling us is that we should ditch those pesky co-authors. If I could write a paper with one graduate student, but a second student also wants to participate, tough cookies. That’s the difference between getting 1/2 a point and 1/3 of a point. Sure, that additional student might improve the paper dramatically. They might also learn a thing or two. But on the other hand, they’ll hurt your rankings.

(Note: currently on CSRankings, graduate students at the same institution don’t get included in the institutional rankings. So including them on your papers will actually reduce your school’s rank.)

I hope it goes without saying that this could create bad incentives.

Second, in fields that mix systems and theory — like computer security — CSRankings is telling us that theory papers (which typically have fewer authors) should be privileged in the rankings over systems papers. This creates both a distortion in the metrics, and also an incentive (for authors who do both types of work) to stick with the one that produces higher rankings. That seems undesirable. But it could very well happen if we adopt these rankings uncritically.

Finally, there’s this focus on “top conferences”. One of our big problems in computer science is that we spend a lot of our time scrapping over a very limited number of slots in competitive conferences. This can be ok, but it’s unfortunate for researchers whose work doesn’t neatly fit into whatever areas those conference PCs find popular. And CSRankings gives zero credit for publishing anywhere but those top conferences, so you might as well forget about that.

(Of course, there’s a question about what a “top conference” even is. In Computer Security, where I work, CSRankings does not consider NDSS to be a top conference. That’s because only three conferences are permitted for each field. The fact that this number seems arbitrary really doesn’t help inspire a lot of confidence in the approach.)

So what can we do about this?

As much as I’d like to ditch rankings altogether, I realize that this probably isn’t going to happen. Nature abhors a vacuum, and if we don’t figure out a rankings system, someone else will. Hell, we’re already plagued by U.S. News, whose methodology appears to involve a popcorn machine and live tarantulas. Something, anything, has to be better than this.

And to be clear, CSRankings isn’t a bad effort. At a high level it’s really easy to use. Even the issues I mention above seem like things that could be addressed. More conferences could be added, using some kind of metric to scale point contributions. (This wouldn’t fix all the problems, but would at least mitigate the worst incentives.) Statistics could perhaps be updated to adjust for graduate students, and soften the blow of having co-authors. These things are not impossible.

And fixing this carefully seems really important. We got it wrong in trusting U.S. News. What I’d like is this time for computer scientists to actually sit down and think this one out before someone imposes a ranking system on top of us. What behaviors are we trying to incentivize for? Is it smaller author lists? Is it citation counts? Is it publishing only in a specific set of conferences?

I don’t know that anyone would agree uniformly that these should be our goals. So if they’re not, let’s figure out what they really are.

13 thoughts on “A few thoughts on CSRankings.org”

  1. Honestly, it was pretty mentally easy to write off CSRankings as irrelevant when I saw that they’re missing almost half of my lab’s faculty.

  2. While this is a fair concern, I actually think CS Rankings is a pretty good effort.

    To highlight something, you should file these as issues against the github project! Or better yet, make a pull request.

    Their goal is to avoid metrics (such as citations), which are easy to game. The idea is that top-tier publications are a good analog and are (somewhat harder) to game than other things. Sure, you could produce lots of “low quality” work; but if you’re getting into conferences with < 30% accept rates, that work is probably decently high quality. And passing the gold standard, peer review.

    I have my own issues with the rankings (which reward "large" departments that can afford to have researchers publish in every field and perform strange normalization between different fields). But they're an earnest, decent effort.

  3. Another point of concern that I don’t think was mentioned: if CSRankings is just an “add up the points” type of ranking, doesn’t that reward bigger institutions of mediocre quality, putting them at the same level as a smaller, more advanced program at an elite university? Or maybe that’s the point – just measuring and rewarding raw output, and not evaluating a percentage of quality.

    If I was shopping for a CS uni, I would want to go to a not-so-large place where a large percentage of the program was highly-regarded and recognized, as opposed to a large program where a smaller percentage of a big pool was doing the heavy lifting.

    Chacun à son goût, I suppose.

    1. They do put the number of faculty right by the score, if that matters to you. They also freely break it down by faculty so you can scroll through and easily see how “top heavy” it is. Reporting averages might seem equally unfair, penalizing for the hiring new young faculty.

      I think CSRankings does a good job of being clear, transparent, accessible and informative. It makes it very easy to answer the question of “who is good, in what, and why?” which may be all we can really ask for a rating.

  5. Interesting but CSRankings is completely useless for computational biology, because it only considers conferences. Virtually all papers in comp. bio. appear in journals. I un-checked all the fields except comp. bio to see what I got, and the rankings were ridiculous – some schools that are unknown in the field showed up because of 1 person who had a conference paper somewhere. And schools that are clearly among the best don’t show up at all. (Aside: this rankings site simply perpetuates a long-standing problem in the field of CS, which is the excessive reliance on conferences. Journal publication is simply better – more rigorous, better edited, better reviewed, etc. That’s why every field of science uses journals except CS.)

    1. This is addressed in their FAQ “CSrankings uses DBLP as its data source, and DBLP does not currently index general science journals (including Science, Nature, and PNAS).” The project is open source and maintained by a single individual. If you could perhaps suggest a way to fix this (solely technical) issue (or better yet, do it and submit a pull request), I’m sure CSRankings will happily add Nature/Science/etc.

    2. Maybe Journal publication is better in your subfield…don’t make a blanket statement that isn’t true for others. In one of my subfields, programming languages, the conferences have consistently been better than journals in terms of both rigor and reviewing, with editing a wash…until recently 3 of the top conferences got together to publish their results in a journal (but otherwise use essentially the same processes they did before). In my other subfield, software engineering, conferences continue to dominate, with at least as much quality and faster turnaround (which is worth quite a lot).

  6. I think that Matt has pointed out many of the problems with this particular ranking method. I would add that many fields (including mine) are inherently collaborative. Simply splitting one point among co-authors (even if restricted to non graduate students) creates a definite disincentive for collaboration between faculty, at least until they have tenure and can ignore this stuff. The ranking method would give a high rank to a large department of prima donnas. I would not want to be in such a department.

    The concept of “top conference” is also most fuzzy and tends to ignore high-impact specialized meetings. In my subfield of robotics, ICRA and IROS are indeed common venues, but other very high prestige venues such as MICCAI and IPCAI are ignored by CSRankings, as are journals. At the least, there should be some effort to include (and weight) multiple venues. Personally, I think that impact is likely better (though imperfectly) measured through citations.

    A good ranking system should also include external awards and honors (society fellows, society prizes, NAE/NAS/NAM memberships, major awards, etc.)

    1. I think the idea is to have an accessible, straightforward and hard-to-game system. If other things are added (such as prizes, even if we had a well-maintained list of them http://jeffhuang.com/best_paper_awards.html is great but lacking), how do we weight them? How much publications do we trade off for a best paper award? Why is that fair?

      No metric is perfect, but the current approach seems to provide a decent analog for what we actually care about. Sure, not everyone publishes in those conferences, but if different departments roughly publish in those venues with similar probability, then the metric would be representative.

      https://github.com/emeryberger/CSrankings/issues/558 is a recent, open issue where MICCAI is proposed to be added. I think this is the great strength of CSrankings, its open, collaborative, quantitative nature, and ability to make a pull-request that will alleviate its weaknesses. This is better than a few-times-a-decade survey conducted by a magazine that no one reads.

Comments are closed.