1

  1. The best result on a given dataset has no trace in the form of research paper or any sort of analysis.

  2. We have no idea whose results were those, or what method was created.

  3. Some competition was held, participants uploaded their resultant output files on leader-board.

  4. When the competition was over, the best performing participant didn't come forward to share their methods.

  5. The organizers wrote a survey paper, where they mentioned this unknown result but mentioned that they had no idea who the authors were, probably in the hope that they would write paper in the future.

  6. After the competition, that participant still hasn't written any paper. The only information available is that survey paper by the organizers with the result in the form of a number and nothing else.

Can this be referenced as the state of the art result or does one have to pick the best scoring entry that has a paper as evidence by its author?

Improve this question

2 Answers 2

Reset to default
2

Yes. You can cite the survey paper as evidence to support your claim.

Improve this answer
4
  • But shouldn't state of art be in the form of a research paper, where we can compare our self with that method? Like we have no idea about the author or even any code. The only existence is dummy name and a result which can be cheated? The result are bit suspious.
    – Stanger Danger
    Commented Aug 21, 2020 at 1:40
  • 1
    If you think they cheated, you should have stated that in the question. My answer assumes the competition was designed to prevent cheating. The truth is what matters; what has been published is only relevant if it adds to that truth.
    – Anonymous Physicist
    Commented Aug 21, 2020 at 2:13
  • I will rephrase my question. Can i use the 2nd best score with a proper research paper as SOTA in my writing and can I argue that, because the best score has no evidence in the form of any research paper or code so I can neglect that, as I don't find much evidence for them to be at the top?
    – Stanger Danger
    Commented Aug 21, 2020 at 2:36
  • 2
    But you do have evidence they are at the top. Just say their methods have not been disclosed.
    – Anonymous Physicist
    Commented Aug 21, 2020 at 2:43
1

Was the competition ranking based on a blinded test set?

  • If not, the none of the results of the competition are very strong evidence.

  • If yes, unless there was gross misconduct* the blinding by the competition provides strong evidence in favor of the best ranked result.

Thus, cite the best result with the publication about the competition. If you are uncomfortable, you can also cite e.g. the 3 best results known so far.

* E.g. the "best" result having been obtained by improper knowledge about the test set reference labels. That would likely cast doubt again on the reliability competition and in turn on its publication.
I'm looking at the competition as something like a ring trial/round robin. The evidence generated depends critically on the trust in the organizer.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .