03. IR - Introduction, Evaluation

The goal of IR is to determine the relevance of a document in terms of a specific topic.

·         Can be anything, web page, word file, text file etc.

A set of documents

Does a document satisfy the information need of the user or does it help complete the user’s task?

How many documents contain the term

how often does the term appear per document

• Exact matching (concatenated with or)

• Boolean queries (and / or / not operators)

• Expanded queries (incorporate synonyms)

• Wildcard queries, phonetic queries, phrase queries

An inverted index allows efficient document retrieval by storing term statistics such as document frequency and term frequency. It avoids reading full documents and enables fast ranking of relevant results.

1. Document Frequency (df) – Number of documents containing a term.

2. Term Frequency (tf) – Number of times a term appears in a document.

3. Posting List – A list of document IDs where the term appears.

4. Document Length – Length of each document.

The Dictionary <T> maps text to T

T is a posting list or potentially other data about the term depending on the index

• Random lookup

• Fast (creation & lookup

• Memory efficient

• Relevance to the need rather than to the query

  • Query is a shorthand for an instance of information need, its initial verbalized presentation by the user

• Relevance is assumed to be a binary attribute

  • A document is either relevant to a query / need or it is not

Number of occurences of term t in document d

TF-IDF helps rank documents by assigning higher scores to rare terms that appear frequently in a document, making them more relevant than common words.

BM25 improves TF-IDF by introducing term frequency saturation and document length normalization, making the retrieval process more effective for long queries.

• Precision measures how many retrieved documents are actually relevant.

• Recall measures how many relevant documents were retrieved out of all possible relevant documents.

MRR focuses on the position of the first relevant document in search results. It calculates the reciprocal of its rank and averages it over multiple queries.

Unlike precision, nDCG considers different levels of relevance (perfect, highly relevant, relevant, irrelevant) and discounts relevance based on document position.

1. Defining Relevance – Different users may perceive relevance differently.

2. Handling Ambiguity – Queries may have multiple interpretations.

3. Limited Judgment Data – Not all documents can be manually assessed.

4. Computational Cost – Some evaluation metrics are expensive to compute.

BM25 modifies TF-IDF by introducing term frequency saturation and document length normalization, improving retrieval effectiveness for long documents and queries.