How to build_a_search_engine
•
0 likes•664 views
This document discusses how to build a small distributed search engine using open source software. It describes the main subsystems of a search engine, including a page database, crawler, parser, indexer and link graph database. It then introduces Apache Hadoop and Apache Lucene as open source tools that can be used to build each subsystem in a distributed manner. Hadoop provides HDFS for distributed storage and MapReduce for distributed processing, while Lucene handles full-text indexing and search. The document outlines how Lucene indexes and searches document contents, and how its components can be integrated with HDFS to build a distributed search index and query system.
Report
Share
How to build_a_search_engine
- 1. How to build a small distributed search engine using open source software
- 2. Building a distributed search engine Search engine subsytems: ● Page database ● List of the pages to retrieve ● Pages retrieval and save ● Page content parsing ● Full-text indexing of the contents ● Graph database of the links for ranking
- 3. Building a distributed search engine Open Source Software • Apache Hadoop • • • • MapReduce HDFS HBase Apache Lucene
- 4. Building a distributed search engine HDFS Hadoop Distributed File System
- 5. Building a distributed search engine HDFS – Assumptions and goals ● Hardware failure ● Big data ● Write once / read many ● Moving computation, not data
- 6. Building a distributed search engine
- 7. Building a distributed search engine
- 8. Building a distributed search engine Lucene
- 9. Building a distributed search engine Lucene - Inverse Indexing Term Doc Id Weight JUG 301 198 120 0.97 0.65 0.43 301 278 451 103 763 0.94 0.15 0.87 0.45 0.77 Lugano
- 10. Building a distributed search engine Lucene - Indexing main classes IndexWriter Directory Analyzer Document Field
- 11. Building a distributed search engine Lucene - Searching main classes IndexSearcher Collector Query TopDocs ScoreDoc
- 12. Building a distributed search engine Lucene - Analyzers StopWords ”the book is on the table” → [book, table] Stemming [paint, paints, painted, …] → paint Synonims [cat, feline] → cat
- 13. Building a distributed search engine Lucene - Search options Fields Wildcards Title: JUG body: ”JUG Lugano” J?G → [JUG, JAG, ...] J*G →[JUG, JEEG, JUNG, …] Fuzzy (basata su vocabolario) JUG~[n] → [MUG, JAG, …]
- 14. Building a distributed search engine Lucene - Search options Range Boost JUG^5 Lugano ”JUG Lugano”^5 Proximity Year: [2002 TO 2012] Name: {Alberto TO Andrea} ”JUG Lugano”~5 Boolean and existance AND, OR, NOT, (), +, -
- 15. Building a distributed search engine HDFS - Lucene Integration File copy from/to HDFS Patch IndexWriter/Director IndexWriter/Directory Rewrite of IndexWriter on RAM Lucene 4
- 16. Building a distributed search engine And now... Hands on!