Search Solutions 2015: Towards a new model of search relevance testing
•Download as ODP, PDF•
0 likes•2,474 views
How and why do we test search relevance? Can we do better? What tools and techniques are available for test-driven relevancy tuning?
Report
Share
Search Solutions 2015: Towards a new model of search relevance testing
- 1. Charlie Hull - Managing Director 20th October 2015 Search Solutions charlie@flax.co.uk www.flax.co.uk/blog +44 (0) 8700 118334 Twitter: @FlaxSearch Towards a new model of test-based relevance tuning
- 2. Building open source search applications since 2001 Independent, honest advice and analysis Expert design & development, Apache Solr committers Test-driven relevancy and performance tuning Custom training & mentoring for your staff Flexible support up to 24/7/365 with SLAs
- 3. Building open source search applications since 2001 Independent, honest advice and analysis Expert design & development, Apache Solr committers Test-driven relevancy and performance tuning Custom training & mentoring for your staff Flexible support up to 24/7/365 with SLAs Come and join the open source search community (tonight?)
- 5. Why bother testing? Throwing it over the fence Some (slightly) better methods A collaborative model Quepid & other tools A better way to test @FlaxSearch
- 6. Search is Magic Search doesn't affect the bottom line The new search engine is better than the old one We can just fix this one problem here... Why bother testing? @FlaxSearch
- 7. Business users / content creators know search is broken Managers tell search developers to 'fix it' Search developers don't understand why it needs fixing Business users / content creators don't understand side effects of a fix Bad communication, internal politics, search gets worse! Throwing it over the fence @FlaxSearch
- 8. Avoiding the HiPPO @FlaxSearch © Copyright William J Bagshaw and licensed for reuse under this Creative Commons Licence
- 9. Identify what to test – Query logs – 'Most valuable' queries – Languages/markets – Segmented query types Keep proper records – Manual query testing – Record relevance judgements • Per page or per result? – Say why – Have an overall score Use the same test system Some (slightly) better methods @FlaxSearch
- 10. Some (slightly) better methods @FlaxSearch
- 11. Problems: – Slow iterations – Lots of error-prone copy-and-paste – Unwieldy – Not really collaborative Improvements: – Build test UIs – Use a better scoring algorithm • e.g. average discounted distance – Bring in other data e.g. web analytics • But remember clicks only 45% predict relevance (75% is achievable) (Susan Dumais, Microsoft) Some (slightly) better methods @FlaxSearch
- 12. In software we use tests to collaborate 1. “What should happen in this case?” 2. Write test code to check TDD can lead to improved software quality Why not with search relevancy? Test-based relevancy @FlaxSearch
- 13. A client approached Flax: – “What's the current state of the art /academically proven way to test relevance?” – Er..... I remembered something called Quepid... So this happened.... @FlaxSearch
- 14. Built by Doug Turnbull of OpenSource Connections A browser-based tool for tuning relevance Needed some development for enterprise use - We did this working with our client Let's take a look... Quepid @FlaxSearch
- 15. You should test your searches in a methodical way Collaboration between 'the business' and developers is vital Some tools now exist to help Hopefully this is the first step to better relevance tuning Conclusions @FlaxSearch
- 16. Thankyou! Any questions? charlie@flax.co.uk www.flax.co.uk/blog +44 (0) 8700 118334 Twitter: @FlaxSearch