Web Crawl Dataset - Legal & Practical feedback [closed]

Question

I'm working on a web site classification method. Papers in field (Text Classification / Web Page Classification) are usually using:

• a) Ancient, but well known & widely adopted, datasets like WebKB 7-sectors
• b) Their own datasets, collected from domains found at a public categorized web directory

What puzzles me:

• after review of recent research, I found that for b) cases, authors are not that happy to make dataset (corpus) publicly available.

They clearly state where they harvested web domains and do provide parameters of the crawl - but, I assume it would make scientific sense to publish exact corpus - so, future comparisons have more solid ground. Yet, I rarely find dataset link in a paper (interestingly enough: in few cases, I recall: authors published the corpus but never mentioned it in the article - I found it accidentally while searching GitHub / Google for PDF of the original source)

Now, I'm worried if there are legal concerns here - that I might not be aware of. I would appreciate very much thoughts & feedback from your practice / experience:

• Should we allow our self to interpret copyright notice (~100% of pages in the sample) "all rights reserved" less rigidly, for our case, case of a scientific research?

  • But how to sleep well if I published gigabytes of web pages, having such copyright notice, on a Data Set repository with Creative Commons or GPLv3 licence? I mean, it is hard to play dumb if ever called out for such "disagreement" between the repository content and the licence.

• Is it about me, or you agree - that it sounds really insane if we should ask for written permission to each of 1400 legal entities (in my current sample), running business web site - being researched?

• how you guys handle this dilemma?

• or would you stick only with ancient/mainstream datasets?

My particular research has focus more toward "web site" than "web page / text" classification aspect. Hence, I have strong motive to crawl an additional dataset (from domains harvested from https://dmoztools.net/), beside using the mainstream ones (since these rarely provide web-site context, and mostly contain web pages / single documents). If I take that methodological path, I have to make it available -- but: are there hidden problems in such path?

Thanks for all thoughts in advance & this is not an easy one :)

Answer 1

There seem to me to be two issues here: copyright and reproducibility of research. You pose them as being in conflict, but I don't think that is the case.

First, in most jurisdictions, publishing someone's copyright work is a violation. "Just for research purposes" probably won't save you in these times, though once it might have done so.

However, the reproducibility issue isn't quite as you pose it here. Suppose I publish an algorithm (I'm assuming it is an algorithm, actually - deterministic). Suppose I also publish the data set on which it is run. If someone else runs my algorithm on my data set does anyone expect that it won't produce the same result. Even a randomized "algorithm" should produce the same results within statistical limits when run on the same data set.

But that isn't really the meaning of reproducible. I make an hypothesis about what should occur in a situation. I test it. Others should be able to test it under similar constraints. But I doubt that my hypothesis was "X happens in Y dataset with Z methodology". More likely Y here is treated as a category of datasets, not a specific one.

So, if you publish the mechanism by which you gather the dataset, rather than the dataset itself (along with Z), then others can test your work and the results will be more meaningful than if you publish the dataset itself. Thus you avoid the copyright problem altogether.

One reason that authors don't want to publish some datasets is to avoid stepping in quagmires such as copyright. Others involve human subject privacy, etc. But, that shouldn't matter if the hypothesis isn't overly tied to a specific dataset, but to a category of similarly gathered datasets.