Tweaking the Base Score: Lucene/Solr Similarities Explained
•
1 like•8,977 views
This talk was given during Activate Conference 2019. Lucene has a lot of options for configuring similarity, and Solr inherits them. Similarity makes the base of your relevancy score: how similar is this document to the query? The default similarity (BM25) is a good start, but you may need to tweak it for your use-case. In this session, you will learn how BM25 works and how you may want to change its parameters. Then, we'll move to other similarity classes: DFR, DFI, IB and LM. You will learn the thinking behind them, how that thinking translates to the similarity score, and which parameters allow you to tweak how score evolves based on things like term frequency or document length. By the end, you’ll have a good understanding of which similarity options are likely to work well for your use-case. You'll know which tunables are available and whether you need to implement a custom similarity class. As an example, we’ll focus on E-commerce, where you often end up ignoring term frequency altogether. Key Takeaway 1) What are the built-in Lucene/Solr similarities and what they do 2) Which similarity to use for which use-case 3) How to use a custom similarity class in Solr Learn more about search relevance and similarity: sematext.com/blog/search-relevance-solr-elasticsearch-similarity
Report
Share
Tweaking the Base Score: Lucene/Solr Similarities Explained
- 2. Tweaking the Base Score: Lucene/Solr Similarities Explained Demo: github.com/sematext/activate/tree/master/2019 More info: sematext.com/blog/search-relevance-solr-elasticsearch-similarity Radu Gheorghe Rafał Kuć www.sematext.com
- 3. Agenda BM25 - Best Match: the default DFR - Divergence From Randomness framework DFI - Divergence From Independence IB - Information-Based models LM - Language Models Custom similarity Putting it all together
- 4. TF*IDF You know, for historical reasons
- 5. BM25 - the TF part freq / (freq + k1 * (1 - b + b * dl / avgdl)) Best for Most 😁
- 6. BM25 tunables freq / (freq + k1 * (1 - b + b * dl / avgdl)) k1 - raise or lower ceiling
- 7. BM25 tunables freq / (freq + k1 * (1 - b + b * dl / avgdl)) doc length normalization
- 8. BM25 demo yes, that’s how we look when we give demos
- 9. BM25 Good default. You can tune the weight of freq and docLength.
- 10. Divergence From Randomness Basic Model G, I(n), I(ne), I(F) After Effect L, B Normalization H1, H2, H3, Z, none
- 11. tf * c * avgFieldLength / docFieldLength Divergence From Randomness - H1
- 12. Divergence From Randomness - H1 No normalization, and H1 with c == 1, 3, 5, 7
- 13. tf * log2 (1 + c * (avgFieldLength / docFieldLength)) Divergence From Randomness - H2
- 14. Divergence From Randomness - H2 No normalization, and H2 with c == 1, 3, 5, 7
- 15. tf * (avgFieldLength / docFieldLength)Z Divergence From Randomness - Z
- 16. Divergence From Randomness - Z No normalization, and Z with z == 0.1, 0.2, 0.3, 0.4
- 17. (tf * mu * ((totalTermFreq + 1) / (#fieldTokens + 1))) (docFieldLength + mu) * mu Divergence From Randomness - H3
- 18. Divergence From Randomness - H3 No normalization, and H3 with mu == 1, 3, 5, 7
- 19. DFR demo Only one, I promise
- 20. DFR Framework. Tunable: choose algorithm and tune parameters for both IDF* and docLength. * generic name for importance of this term
- 21. Divergence From Independence expected frequency
- 22. Divergence From Independence docLength*totalTermFrequency/numberOfFieldTokens expected frequency
- 23. DFI: Standardized (actual - expected)/sqrt(expected)
- 24. DFI demo Oh, but don’t remove stopwords*! 1) arbitrarily chops field length 2) stopwords aren’t always stopwords ;)
- 25. DFI Simple. Parameterless. Flexible: works well with various datasets.
- 26. Information Based how much information we get from this term?
- 27. Information Based Distribution Log-Logistic, Smoothed Power-Law Lambda DF, TTF Normalization H1, H2, H3, Z, none
- 28. Information Based - Log-Logistic log( tfn / (lambda + 1) )
- 29. Information Based - Log-Logistic lambda: 0.1 (red), 0.3 (black), 0.8 (blue)
- 30. Information Based - Retrieval Function the average of the document information brought by each query term
- 31. Information Based - Retrieval Function - DF number of matching documents (docFrequency + 1) / (numberOfDocuments + 1)
- 32. Information Based - Retrieval Function - TTF total number of term occurrences (totalTermFrequency + 1) / (numberOfDocuments + 1)
- 33. IB demo
- 34. IB Framework. like DFR. Even has the same normalization options. But newer and, in the paper, better.
- 35. Language Models probability of a term being our term
- 36. Language Models totalTermFreq/totalFieldTokens probability of a term being our term
- 37. Language Models: Jelinek-Mercer log( (1-λ)* tf )docLength λ * probability
- 39. LM Two probabilistic models. Similar approach to DFI, but tunable.
- 40. Custom Similarity compute a similarity score using custom code
- 41. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 42. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 43. Custom Similarity - Activate Similarity Factory public class ActivateSimilarityFactory extends SimilarityFactory { private volatile Similarity similarity; public void init(SolrParams params) { super.init(params); } public Similarity getSimilarity() { if (similarity == null) { similarity = new ActivateSimilarity(); } return similarity; } }
- 44. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 45. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 46. Custom Similarity - Similarity public class ActivateSimilarity extends Similarity { public ActivateSimilarity() {} public long computeNorm(FieldInvertState state) { return 1; } public Similarity.SimScorer scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) { return new ActivateSimScorer(); } }
- 47. Custom Similarity - SimScorer public class ActivateSimScorer extends Similarity.SimScorer { public float score(float freq, long norm) { return freq; } }
- 48. Custom Similarity - SimScorer public class ActivateSimScorer extends Similarity.SimScorer { public float score(float freq, long norm) { return freq; } }
- 50. Custom When you need something special, like disregarding term frequency.
- 53. THANK YOU