Information Retrieval 02
- 1. Introduction to Information RetrievalIntroduction to Information Retrieval
Introduction to
Information Retrieval
Scoring, Term Weighting and the Vector Space Model
Slides from Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze
1Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 2. Introduction to Information RetrievalIntroduction to Information Retrieval
Outlines
Ranked retrieval
Scoring documents
Term frequency
Collection statistics
Weighting schemes
Vector space scoring
2Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 3. Introduction to Information RetrievalIntroduction to Information Retrieval
Boolean Retrieval - AND/OR/NOT
A B
All documents
C
3
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 4. Introduction to Information RetrievalIntroduction to Information Retrieval
Boolean Retrieval
Weights assigned to terms are either “0” or “1”
“0” represents “absence”: term isn’t in the document
“1” represents “presence”: term is in the document
Build queries by combining terms with Boolean
operators
AND, OR, NOT
The system returns all documents that satisfy the
query
4Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 5. Introduction to Information RetrievalIntroduction to Information Retrieval
Problem with Boolean search: feast or famine
Boolean queries often result in either too few (=0) or
too many (1000s) results.
Query 1: “standard user dlink 650” → 200,000 hits
Query 2: “standard user dlink 650 no card found”: 0
hits
It takes a lot of skill to come up with a query that
produces a manageable number of hits.
AND gives too few; OR gives too many
5Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 6. Introduction to Information RetrievalIntroduction to Information Retrieval
Feast or famine: not a problem in ranked retrieval
When a system produces a ranked result set,
large result sets are not an issue
Indeed, the size of the result set is not an issue
We just show the top k ( ≈ 10) results
We don’t overwhelm the user
Premise: the ranking algorithm works
Today’s question: Given a large list of documents that
match a query, how to order them according to their
relevance?
Answer: Scoring the documents
6
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 7. Introduction to Information RetrievalIntroduction to Information Retrieval
Scoring as the basis of ranked retrieval
We wish to return in order the documents most
likely to be useful to the searcher
How can we rank-order the documents in the
collection with respect to a query?
Assign a score – say in [0, 1] – to each document
This score measures how well document and query
“match”.
Let us say we have a one-term query – How will you
Compute the score ?
7Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 8. Introduction to Information RetrievalIntroduction to Information Retrieval
Query-document matching scores
We need a way of assigning a score to a
query/document pair
Let’s start with a one-term query
If the query term does not occur in the document:
score should be 0
The more frequent the query term in the document,
the higher the score (should be)
What are the two issues with the proposed approach
8Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 9. Introduction to Information RetrievalIntroduction to Information Retrieval
Ranking based on raw term frequency - issues
All terms have equal weight
Larger/Bigger documents have more terms and thus
the score is larger
N times more frequency/occurrence does not mean
N times more relevant
9Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 10. Introduction to Information RetrievalIntroduction to Information Retrieval
Jaccard coefficient
A commonly used measure of overlap of two sets A
and B
What is jaccard(A,B) =
jaccard(A,B) = |A ∩ B| / |A ∪ B|
jaccard(A,A) = 1
jaccard(A,B) = 0 if A B =∩ 0
A and B don’t have to be the same size.
Always assigns a number between X and Y and why?
Always assigns a number between 0 and 1 because
cause at most the intersection can be as large as the union.
10Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 11. Introduction to Information RetrievalIntroduction to Information Retrieval
Jaccard coefficient: Scoring example
What is the query-document match score that the
Jaccard coefficient computes for each of the two
documents below?
Query: ides of march
Document 1: caesar died in march
Document 2: the long march
11Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 12. Introduction to Information RetrievalIntroduction to Information Retrieval
Jaccard Example
Query: ides of march
Document 1: caesar died in march
Document 2: the long march
A = {ides, of, march}
B1 = {caesar, died, in, march}
B2 = {the, long, march}
6/11/1
},,,,,{1
}{1
=
=
=
BABA
marchofinidesdiedcaesarBA
marchBA
AI B2 ={march}
AUB2 ={ides,long,march,of ,the}
AI B2/ AUB2 =1/5
12Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 13. Introduction to Information RetrievalIntroduction to Information Retrieval
Issues with Jaccard for scoring
It doesn’t consider term frequency (how many times
a term occurs in a document)
Rare terms in a collection are more informative than
frequent terms. Jaccard doesn’t consider this
information
We need a more sophisticated way of normalizing
for length.
Biased towards shorter documents.
13Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 14. Introduction to Information RetrievalIntroduction to Information Retrieval
14
The problem with the first example was that document
2 “won” because it was shorter, not because it was a
better match. We need a way to take into account
document length so that longer documents are not
penalized in calculating the match score.
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 15. Introduction to Information RetrievalIntroduction to Information Retrieval
Binary term-document incidence matrix
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
Each document is represented by a binary vector {0,1}∈ |V
15Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 16. Introduction to Information RetrievalIntroduction to Information Retrieval
Term-document count matrices
Consider the number of occurrences of a term in a
document:
Each document is a count vector in ℕv
: a column below
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 157 73 0 0 0 0
Brutus 4 157 0 1 0 0
Caesar 232 227 0 2 1 1
Calpurnia 0 10 0 0 0 0
Cleopatra 57 0 0 0 0 0
mercy 2 0 3 5 5 1
worser 2 0 1 1 1 0
16Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 17. Introduction to Information RetrievalIntroduction to Information Retrieval
Bag of words model
Vector representation doesn’t consider the ordering
of words in a document
John is quicker than Mary and Mary is quicker than
John have the same vectors
This is called the bag of words model.
17Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 18. Introduction to Information RetrievalIntroduction to Information Retrieval
Term frequency tf
The term frequency tft,d of term t in document d is
defined as the number of times that t occurs in d.
We want to use tf when computing query-document
match scores. But how?
Raw term frequency is not what we want
A document with 10 occurrences of the term is more
relevant than a document with 1 occurrence of the term.
But not 10 times more relevant.
Relevance does not increase proportionally with
term frequency.
18Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 19. Introduction to Information RetrievalIntroduction to Information Retrieval
Log-frequency weighting
The log frequency weight of term t in d is
0 → 0, 1 → 1, 2 → 1.3, 10 → 2, 1000 → 4, etc.
How will you compute Score for a document-query pair:
sum over terms t in both q and d:
score
The score is 0 if none of the query terms is present in the
document.
>+
=
otherwise0,
0tfif,tflog1 10 t,dt,d
t,dw
∑ ∩∈
+= dqt dt )tflog(1 ,
19Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 20. Introduction to Information RetrievalIntroduction to Information Retrieval
Document frequency
Rare terms are more informative than frequent terms
stop words (a, and, the…..) are not informative
Consider a term in the query that is rare in the
collection (e.g., arachnocentric)
A document containing this term is very likely to be
relevant to the query arachnocentric
→ We want a high weight for rare terms like
arachnocentric.
20Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 21. Introduction to Information RetrievalIntroduction to Information Retrieval
Document frequency, continued
Frequent terms are less informative than rare terms
Consider a query term that is frequent in the
collection (e.g., high, increase, line)
A document containing such a term is more likely to
be relevant than a document that doesn’t
But it’s not a sure indicator of relevance.
→ For frequent terms, we want high positive weights
for words like high, increase, and line
But lower weights than for rare terms.
We will use document frequency (df) to capture this.
21Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 22. Introduction to Information RetrievalIntroduction to Information Retrieval
idf weight
dft is the document frequency of t: the number of
documents that contain t
dft is an inverse measure of the informativeness of t
What can be the max value of dft
dft ≤ N
We define the idf (inverse document frequency) of t
by
Why We use log (N/dft) instead of N/dft
to “dampen” the effect of idf.
)/df(logidf 10 tt N=
Will turn out the base of the log is immaterial.
22Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 23. Introduction to Information RetrievalIntroduction to Information Retrieval
23
Mathematically the base of the log function does not
matter and constitutes a constant multiplicative factor
towards the overall result.
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 24. Introduction to Information RetrievalIntroduction to Information Retrieval
idf example, suppose N = 1 million
term dft idft
calpurnia 1 ?
animal 100 ?
sunday 1,000 ?
fly 10,000 ?
under 100,000 ?
the 1,000,000 ?
There is one idf value for each term t in a collection.
)/df(logidf 10 tt N=
24Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 25. Introduction to Information RetrievalIntroduction to Information Retrieval
idf example, suppose N = 1 million
term dft idft
calpurnia 1 6
animal 100 4
sunday 1,000 3
fly 10,000 2
under 100,000 1
the 1,000,000 0
There is one idf value for each term t in a collection.
)/df(logidf 10 tt N=
25Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 26. Introduction to Information RetrievalIntroduction to Information Retrieval
Effect of idf on ranking
Does idf have an effect on ranking for one-term
queries, like
iPhone
idf has no effect on ranking one term queries
idf affects the ranking of documents for queries with at
least two terms
For the query capricious person, idf weighting makes
occurrences of capricious count for much more in the final
document ranking than occurrences of person.
26Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 27. Introduction to Information RetrievalIntroduction to Information Retrieval
27
So for a term one query, you're going to have one of these terms of N
over the document frequency, and it'll be worked out. But it's going to
be just a scaling factor. Which, since there's only one IDF value for each
term will be applied to every document, and therefore, it won't affect the
ranking in any way.
You only get an effect from IDF when you have multiple-terms in a
query. So, for example, if we have the query, capricious person, well,
now, we're in a situation where capricious is a much rarer word. And so
IDF will say. Pay much more attention to documents that contain the
word capricious, than to documents that contain just the word person in
ranking your retrieval results.
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 28. Introduction to Information RetrievalIntroduction to Information Retrieval
Collection vs. Document frequency
The collection frequency of t is the number of
occurrences of t in the collection, counting
multiple occurrences.
Example:
Which word is a better search term (and should
get a higher weight)?
Word Collection frequency Document frequency
insurance 10440 3997
try 10422 8760
What does it mean
28Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 29. Introduction to Information RetrievalIntroduction to Information Retrieval
tf-idf weighting
The tf-idf weight of a term is the product of its tf
weight and its idf weight.
Best known weighting scheme in information retrieval
Note: the “-” in tf-idf is a hyphen, not a minus sign!
Alternative names: tf.idf, tf x idf
Increases with the number of occurrences within a
document
Increases with the rarity of the term in the collection
)df/(log)tf1log(w 10,, tdt Ndt
×+=
29Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 30. Introduction to Information RetrievalIntroduction to Information Retrieval
Score for a document given a query
There are many variants
How “tf” is computed (with/without logs)
Whether the terms in the query are also weighted
…
30
Score(q,d)= tf.idft,dt∈q∩d
∑
Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 31. Introduction to Information RetrievalIntroduction to Information Retrieval
Binary → count → weight matrix
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 5.25 3.18 0 0 0 0.35
Brutus 1.21 6.1 0 1 0 0
Caesar 8.59 2.54 0 1.51 0.25 0
Calpurnia 0 1.54 0 0 0 0
Cleopatra 2.85 0 0 0 0 0
mercy 1.51 0 1.9 0.12 5.25 0.88
worser 1.37 0 0.11 4.15 0.25 1.95
Each document is now represented by a real-valued
vector of tf-idf weights ∈ R|V|
31Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 32. Introduction to Information RetrievalIntroduction to Information Retrieval
Documents as vectors
So we have a |V|-dimensional vector space
Terms are axes of the space
Documents are points or vectors in this space
Very high-dimensional: tens of millions of dimensions
when you apply this to a web search engine
These are very sparse vectors - most entries are zero.
32Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 33. Introduction to Information RetrievalIntroduction to Information Retrieval
Queries as vectors
Key idea 1: Do the same for queries: represent them
as vectors in the space
Key idea 2: Rank documents according to their
proximity to the query in this space
proximity = similarity of vectors
proximity ≈ inverse of distance
Recall: We do this because we want to get away
from the you’re-either-in-or-out Boolean model.
Instead: rank more relevant documents higher than
less relevant documents
33Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 34. Introduction to Information RetrievalIntroduction to Information Retrieval
Formalizing vector space proximity
First cut: distance between two points
( = distance between the end points of the two vectors)
Euclidean distance?
Euclidean distance is a bad idea . . .Why?
. . . because Euclidean distance is large for vectors of
different lengths.
34Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 35. Introduction to Information RetrievalIntroduction to Information Retrieval
Why distance is a bad idea
The Euclidean
distance between q
and d2 is large even
though the
distribution of terms
in the query q and the
distribution of
terms in the
document d2 are
very similar.
35Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 36. Introduction to Information RetrievalIntroduction to Information Retrieval
Use angle instead of distance
Thought experiment: take a document d and append
it to itself. Call this document d .′
“Semantically” d and d have the same content′
The Euclidean distance between the two documents
can be quite large
The angle between the two documents is 0,
corresponding to maximal similarity.
Key idea: Rank documents according to angle with
query.
36Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 37. Introduction to Information RetrievalIntroduction to Information Retrieval
From angles to cosines
.
Rank documents in decreasing order of the angle between
query and document – A minor issue?
Rank documents in increasing order of
cosine(query,document) – Why Cosine?
Cosine is a monotonically decreasing function for the
interval [0o
, 180o
]
37Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 38. Introduction to Information RetrievalIntroduction to Information Retrieval
From angles to cosines
But how – and why – should we be computing cosines?
38Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 39. Introduction to Information RetrievalIntroduction to Information Retrieval
cosine(query,document)
∑∑
∑
==
=
=•=
•
=
V
i i
V
i i
V
i ii
dq
dq
d
d
q
q
dq
dq
dq
1
2
1
2
1
),cos(
Dot product Unit vectors
qi is the tf-idf weight of term i in the query
di is the tf-idf weight of term i in the document
cos(q,d) is the cosine similarity of q and d … or,
equivalently, the cosine of the angle between q and d.
39Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 40. Introduction to Information RetrievalIntroduction to Information Retrieval
Cosine similarity illustrated
40Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 41. Introduction to Information RetrievalIntroduction to Information Retrieval
Cosine similarity exercises
Exercise: Rank the following by decreasing cosine
similarity:
Two docs that have only frequent words (the, a, an, of) in
common.
Two docs that have no words in common.
Two docs that have many rare words in common
(wingspan, tailfin).
41Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 42. Introduction to Information RetrievalIntroduction to Information Retrieval
Which documents are similar according to VSM
auto
engine
bonnet
tyres
lorry
boot
car
emissions
hood
make
model
trunk
make
hidden
Markov
model
emissions
normalize
D1 D2 D3
Which documents are semantically similar
42Department of CSE,TSSOT,AUS, SILCHAR11/28/17
- 43. Introduction to Information RetrievalIntroduction to Information Retrieval
Synonymy and Polysemy
auto
engine
bonnet
tyres
lorry
boot
car
emissions
hood
make
model
trunk
make
hidden
Markov
model
emissions
normalize
Synonymy
Will have small
cosine but are
related
Polysemy
Will have large
cosine but not truly
related
43Department of CSE,TSSOT,AUS, SILCHAR11/28/17
Editor's Notes
- Cf. our discussion of how Westlaw Boolean queries didn’t actually outperform free text querying
- 6 4 3 2 1 0
- 6 4 3 2 1 0
- Why do you get these numbers?
Suggests df is better.
- See Law of Cosines (Cosine Rule) wikipedia page