Skip to main content
Law

Return to Question

Clarification and tags
Source Link
feetwet ♦
feetwet ♦
  • 21.9k
  • 13
  • 85
  • 177

I am not even sure whether my question should be here or some other Stack Exchange platform. Anyway, here goes.

I was talking to my friends a few days ago about search engines and we started discussing whether or not they need permission from websites to crawl them. I looked this one up and it said obon Quora that you don't really need permission to crawl them since they are present in public domain and are public property. If the websites want privacy, they can change their settings appropriately.

However, later on, I talked to a few other people who said that search engines like Google and Bing don't just crawl all websites. They only crawl websites which are registered on their SEO or are on their radar and ranking--Iranking–I didn't really understand this part very much--butmuch–but that doesn't make much sense either since they'd still need to crawl other websites to be updated.

My pointquestion is, if: If I had a working search engine whichthat worked roughly like Google and, Bing, etc. do--on meta tags, etc.--coulddo could I just start crawling and displaying results, or would I need get special permission from the trillions of websites out there before I can actually userun my search engine? Please clarify this for me.

I am not even sure whether my question should be here or some other Stack Exchange platform. Anyway, here goes.

I was talking to my friends a few days ago about search engines and we started discussing whether or not they need permission from websites to crawl them. I looked this one up and it said ob Quora that you don't really need permission to crawl them since they are present in public domain and are public property. If the websites want privacy, they can change their settings appropriately.

However, later on, I talked to a few other people who said that search engines like Google and Bing don't just crawl all websites. They only crawl websites which are registered on their SEO or are on their radar and ranking--I didn't really understand this part very much--but that doesn't make much sense either since they'd still need to crawl other websites to be updated.

My point is, if I had a working search engine which worked roughly like Google and Bing, etc. do--on meta tags, etc.--could I just start crawling and displaying results, or would I need get special permission from the trillions of websites out there before I can actually use my search engine? Please clarify this for me.

I was talking to my friends a few days ago about search engines and we started discussing whether or not they need permission from websites to crawl them. I looked this one up and it said on Quora that you don't really need permission to crawl them since they are present in public domain and are public property. If the websites want privacy, they can change their settings appropriately.

However, later on, I talked to a few other people who said that search engines like Google and Bing don't just crawl all websites. They only crawl websites which are registered on their SEO or are on their radar and ranking–I didn't really understand this part very much–but that doesn't make much sense either since they'd still need to crawl other websites to be updated.

My question is: If I had a search engine that worked roughly like Google, Bing, etc., do could I just start crawling and displaying results, or would I need get special permission from the trillions of websites out there before I can actually run my search engine?

Source Link
user17346
user17346

Do search engines need permission to start searching?

I am not even sure whether my question should be here or some other Stack Exchange platform. Anyway, here goes.

I was talking to my friends a few days ago about search engines and we started discussing whether or not they need permission from websites to crawl them. I looked this one up and it said ob Quora that you don't really need permission to crawl them since they are present in public domain and are public property. If the websites want privacy, they can change their settings appropriately.

However, later on, I talked to a few other people who said that search engines like Google and Bing don't just crawl all websites. They only crawl websites which are registered on their SEO or are on their radar and ranking--I didn't really understand this part very much--but that doesn't make much sense either since they'd still need to crawl other websites to be updated.

My point is, if I had a working search engine which worked roughly like Google and Bing, etc. do--on meta tags, etc.--could I just start crawling and displaying results, or would I need get special permission from the trillions of websites out there before I can actually use my search engine? Please clarify this for me.