Is it legal to use Web crawler for data gathering?

Question

I live in Uk, I am programmer and I have built a personal system that goes to some foreign housing web sites and extracts data from adverts going add by add (using crawler/scraper/web spider), I then process this data and do some analysis to find me a 'good deals'. Since system has already served it purpose for myself I was thinking of making this analytical data public. Basically making a web site and continuously posting results there. In my understanding data that I am gathering is publicly available so I can use that data in whichever way I want. Another is that there is no way to stop me or claim damages as they would be impossible to estimate.

What are legal implications of using Web crawler for data gathering?

Answer 1

You would need to see a lawyer for a definitive answer - generally, if it is on the web and not protected by action requiring acknowledgement of terms of service/use it is "fair game". A greyer area would be if your bot follows the "Robots Exclusion standard" (ie honors the contents of robots.txt file)

Relatedly it may also depend on the amount of data you are spidering - if you spider to quickly it could be construed as a DoS attack, and get you in to hot water (or used as an excuse to get you in to hot water).

In most places you can't copyright a fact - so you are OK on that score - however if you present a layout similar to the original site you take the data from (though it sounds like thats not the case here), you could be in trouble for copyright infringement.