Can I challenge a paper already published in a peer reviewed conference/journal?

Question

I recently came across several papers in the area of Information Extraction and Data Mining which seem to have passed the review process even if they (1) use some new terminology without describing it, (2) replicate some old and popular technique and just describe it in a new way, (3) use new datasets without releasing them publicly or describing them in detail, (4) confusing language that doesn't seem to describe the methods... and much more.

Why challenge them? Such papers give these people a kind of leverage in situations where number of publications count rather than quality of genuine work. They also encourage, let's say, 'pseudoscience' within the scientific community - or fraud, as those more experienced among you say.

I've faced situations like:-

• Someone hired me from a freelance site to implement a previously published paper. The paper doesn't adequately describe the novel methods they claim. I spend a week on it but don't get paid.
• Similar situation but the implementation's accuracy doesn't match the claim.
• In a semi-academic gathering (tea time in a summer school), a discussion turned into a debate. Though I wasn't involved in it, one of the others said something which I pointed out as being logically wrong. It turned personal and one of them claimed that they knew better because of having twice the number of papers than me. Later when I read their papers I found that they didn't contain any ideas that would count as a genuine research.

I guess some of you might have been in similar situations before. Yes it doesn't matter in the longer term; but it is hard to avoid some situations like this and I think it might be hampering the research community as a whole.

Here are the exact points of my question:

1. Should we confront such publications? Would we get anything out of it?
2. Is it even feasible?
3. How can I do so? What actions are possible in such cases? What is the best course of action?

p.s. Yes citations matter more than just throwing papers - but such publications also get citations from other equally content-less papers published in similar conferences.

pps. Most of these papers are from less-than-tier-3 level conferences but still published by top publishers like IEEE and Springer.

Answer 1

I think you have to further examine your motivation for wanting to “challenge” these substandard papers. All of the following are reasons which are suggested in your question.

1. You feel that contentless or decisively problematic publications bring down the quality of the academic community as a whole.
2. You feel that the people who are publishing these works get too much credit.
3. In particular, they are holding themselves up as superior to you.
4. Specific people – e.g. you – suffer when you try to rely on publications which do not do what they promise.

This is a lot to answer, but I’ll take a stab at it:

1. This is a disturbing trend in the world of academic publication. I now get more “journal spam” emails than all other spam put together (well, counting only what makes it into my inbox). One can no longer assume that an academic publication is reputable, legitimate, or sincere in any way without having prior specific knowledge of the people and institutions involved. I think this is a terrible problem, but it's well beyond my pay grade. I do not have a solution in mind.

2. The best revenge is living well. In line with point 1, there is an increasing segment of the worldwide academic community in which people, in order to maintain their very low-paying academic jobs, simply must meet a certain quota of authorship. No one cares about the content or quality; in many cases there is no infrastructure there for them to evaluate the content if they did care. This is very sad. At an individual level, what seems best to me is just to get out of this part of the academic community. In many cases this is best accomplished by relocating to a different part of the world where the academic infrastructure exists. You are not going to get a job at a research university in the United States by having 50 bullshit papers: that's just not the way things work here. However, if you are a scholar in India, you can get offered a job in the United States (or Europe or Japan or...) on the basis of your work: it just has to actually be very good.

3. The person who tells you that he’s right and you’re wrong because he has twice as many papers as you is on the one hand an asshole. On the other hand, he’s splashing around in a pond that you want to get out of: I can assure you that there are many academic environments in which who's right is not decided by counting publications. You sound like a serious person: get yourself into one of these environments.

4. This seems to open up another can of worms, but relative to point 1 above: you simply cannot uncritically take publications at face value. To be honest about it, you can do this to a greater or lesser degree according to the level of trust and esteem you place in the author, the journal, the institution...but doing this at all is a compromise we make only because we are finite beings living in an imperfect world. Learn to develop a “nose” for what is correct and what is problematic. Definitely do not accept a work-for-pay situation which is predicated on the assumption that an unvetted academic work does everything that it promises!

   In general, I would say that point 4 is a good reason to challenge papers: if you sstumbled by relying on a paper or can imagine another serious, reputable person doing the same, then you owe it to the community to try to get the word out about the defects of the work. But as above you should also understand that part of being a good academic is knowing what looks good and what looks bad. There is an ever-expanding sewage sea of worse-than-useless academic papers. You can't take responsibility for getting rid of all of them. But you can teach yourself to avoid the unpleasant experience of being submerged in this sewage sea, and you can teach your colleagues – and, as you get older, your mentees – how to do the same.

Good luck.

Answer 2

I will add my different perspective on the OP's question. @PeteL.Clark answer pretty much covers completely the one side of the story: The OP has discovered some legitimate inconsistencies / mistakes on published papers and wants to make thing rights for the community.

But as we know there is always a second version to each story. Let's discuss some of those aspects as well.

Someone hired me from a freelance site to implement a previously published paper. The paper doesn't adequately describe the novel methods they claim. I spend a week on it but don't get paid.

This does not prove that the paper does something wrong. The assumption that one programmer can implement a method described on one paper that may build on several other papers within a week, is unrealistic / naive. CS scientists may spend 3-12 months working on a paper that may build on previous works that required several other months to implement. A CS paper is not a standalone piece of engineering that needs to describe everything. If it is based on previous works, it only needs to say what it does in addition to previous works. Therefore, in order to fully implement a method described in a paper, you also need to implement all previous methods this paper builds upon. And this takes months to years for newcomers to the suggested area, even if they are good programmers. In other words: Being a good programmer does not guarantee that you can implement every CS scientific paper there is.

Similar situation but the implementation's accuracy doesn't match the claim.

Again that does not guarantee that the paper has done something wrong. Perhaps the OP has done a wrong implementation of the created method. I find very strange the fact that the OP, assumes first that everyone else is wrong and has total confidence in his work, abilities and implementation, even if they contradict published work by many other authors. Assuming everyone else is wrong and my code / implementation is always correct, is not only a characteristic of cranks but contradicts the basic principle of software engineering: All codebases have bugs and first you should look for bugs in your codebase and then look for bugs elsewhere.

...which I pointed out as being logically wrong. The guy turned personal and claimed that he knew better because he had twice the number of papers than me. And later when I read his papers I found that the papers din't contain any ideas that would count as a genuine research.

What does logically wrong means? Did you manage to do experiments and prove things over a tea table? Good for you. And what is this? "The papers didn't contain any ideas that would count as a genuine research". What is genuine research? Is this another version of the No true Scotsman argument? If I do not like someone else's research, I can belittle it by saying that it is no genuine research?

Use new datasets without releasing them publicly or describing them in detail

Again, this is the standard argument of newcomers. "Freely release your code and your datasets so I can use them but I will keep my code by myself so nobody steals it". The main argument is "You have already published your work and I have not, so you must help me". Again, this line of argumentation is wrong. An author has the exclusive right to share or not share his code or datasets. And has the OP asked the authors to give him access to their code or datasets? The fact that someone does not release the code publicly does not mean that if contacted, he will not share.

Conclusively, in most CS PHD students meetings I have encountered, there is always the "purist" guy who insists that he is only interested in A* conference publications, he only wants to do top publications, he is working 2 or more years in his "seminal" paper, most works of his peers is mediocre (as the OP described "not genuine research") and he will never work on incremental papers, since publishing something less than perfect is not suited for him. Unless this guy is exceptionally talented (which is easy to tell by his previous publications) this is also the guy that needs 7 years to get his PHD (if he actually gets one) and has an average h-index of 2, when finishing. Please, do not be that guy (unless you are that talented). Focus on publishing your original work, instead of trying to find errors on everyone else.

Answer 3

A somewhat "indirect" challenge is to is to publish your comments on a website such as PubPeer. It has been historically used to comment mostly on papers in the area of medical research, but it also hosts discussions regarding papers in computer science. A full list of the journals that have comments regarding specific papers is available on that site as well.

In case these discussions point to substantial flaws in a paper, they can be forwarded to the journal editor/publisher.

Answer 4

To answer the question from a more technical point of view: Some journals allow you to publish comments on papers (though I learnt from the comments that the prevalence of this practice varies strongly depending on the field). They usually are like regular publications, having citations, figures, tables and whatever you need and being subject to peer review. Depending on the size and relevance of the criticised paper, such a publication might also be a regular paper.

In your example where a method’s accuracy didn’t match the claim, you can probably publish a comment on this, substantiating your opposing claim with data or similar. Something similar goes for methods that are identical or very similar to existing ones without crediting them¹. However, I consider it unlikely that a journal will publish a comment on the inadequateness of a method’s description. In all cases, it might be sending an outline of your comment to the journal and ask if they would publish it.

Now, given the huge amount of bad papers out there, you could spend all your life writing comments exposing fundamental flaws in papers in your field. Unfortunately, there is no financial and little academic credit for doing this, so it is rarely done. However, if you already spent some time on trying to make use of a paper, you may already have most of what you need to publish such a comment and you may consider it worth your time.

Some modern journals (e.g., Plos One) also have online interfaces for instant comments that allow for criticising a paper with much less effort.

---

^{¹ Consider the possibility of an independent discovery though.}