Take the query "word1 word2" as an example.
BOOLEAN mode indicates that either does your entire query matchmatches the document (e.g. it contains both word1 AND word2). Boolean mode is a strict match.
The formula normally used is based on the Vector Space Model of searching. Very simplified, it figures out two measures to determine how important a word is to a query. The term frequency (terms that occur often in a document are more important than other terms) and the inverse document frequency (a term that occurs in many documents is weighted lower than a term that occurs in few documents). This is known as tf-idf, and is used as a basis for the vector space model. These scores form the basis for the Vector Space Model, which someone else can explain thoroughly. :)