This notebook demonstrates retrieval using PyTerrier on the MSMARCO Passage Ranking corpus.
About the corpus: A passage ranking task based on a corpus of 8.8 million passages released by Microsoft, which should be ranked based on their relevance to questions. Also used by the TREC Deep Learning track.
#!pip install -q python-terrier
import pyterrier as pt
if not pt.started():
pt.init()
from pyterrier.measures import *
dataset = pt.get_dataset('msmarco_passage')
Terrier's default Porter stemming, and stopwords removed.
bm25_terrier_stemmed = pt.BatchRetrieve.from_dataset('msmarco_passage', 'terrier_stemmed', wmodel='BM25')
dph_terrier_stemmed = pt.BatchRetrieve.from_dataset('msmarco_passage', 'terrier_stemmed', wmodel='DPH')
Terrier's default Porter stemming, and stopwords removed. Text is also saved in the MetaIndex to facilitate BERT-based reranking.
import onir_pt
# Lets use a Vanilla BERT ranker from OpenNIR. We'll use the Capreolus model available from Huggingface
vanilla_bert = onir_pt.reranker('hgf4_joint', ranker_config={'model': 'Capreolus/bert-base-msmarco', 'norm': 'softmax-2'})
bm25_terrier_stemmed_text = pt.BatchRetrieve.from_dataset('msmarco_passage', 'terrier_stemmed_text', wmodel='BM25', metadata=['docno', 'text'])
bm25_bert_terrier_stemmed_text = bm25_terrier_stemmed_text >> vanilla_bert
Terrier index using docT5query. Porter stemming and stopword removal applied
bm25_terrier_stemmed_docT5query = pt.BatchRetrieve.from_dataset('msmarco_passage', 'terrier_stemmed_docT5query', wmodel='BM25')
Terrier index using DeepCT. Porter stemming and stopword removal applied
bm25_terrier_stemmed_deepct = pt.BatchRetrieve.from_dataset('msmarco_passage', 'terrier_stemmed_deepct', wmodel='BM25')
43 topics used in the TREC Deep Learning track Passage Ranking task, with deep judgements
pt.Experiment(
[bm25_terrier_stemmed, dph_terrier_stemmed, bm25_terrier_stemmed_text, bm25_bert_terrier_stemmed_text, bm25_terrier_stemmed_docT5query, bm25_terrier_stemmed_deepct],
pt.get_dataset('msmarco_passage').get_topics('test-2019'),
pt.get_dataset('msmarco_passage').get_qrels('test-2019'),
batch_size=200,
filter_by_qrels=True,
eval_metrics=[RR(rel=2), nDCG@10, nDCG@100, AP(rel=2)],
names=['bm25_terrier_stemmed', 'dph_terrier_stemmed', 'bm25_terrier_stemmed_text', 'bm25_bert_terrier_stemmed_text', 'bm25_terrier_stemmed_docT5query', 'bm25_terrier_stemmed_deepct'])
43 topics used in the TREC Deep Learning track Passage Ranking task, with deep judgements
pt.Experiment(
[bm25_terrier_stemmed, dph_terrier_stemmed, bm25_terrier_stemmed_text, bm25_bert_terrier_stemmed_text, bm25_terrier_stemmed_docT5query, bm25_terrier_stemmed_deepct],
pt.get_dataset('trec-deep-learning-passages').get_topics('test-2020'),
pt.get_dataset('trec-deep-learning-passages').get_qrels('test-2020'),
batch_size=200,
filter_by_qrels=True,
eval_metrics=[RR(rel=2), nDCG@10, nDCG@100, AP(rel=2)],
names=['bm25_terrier_stemmed', 'dph_terrier_stemmed', 'bm25_terrier_stemmed_text', 'bm25_bert_terrier_stemmed_text', 'bm25_terrier_stemmed_docT5query', 'bm25_terrier_stemmed_deepct'])
6800 topics with sparse judgements
pt.Experiment(
[bm25_terrier_stemmed, dph_terrier_stemmed, bm25_terrier_stemmed_text, bm25_bert_terrier_stemmed_text, bm25_terrier_stemmed_docT5query, bm25_terrier_stemmed_deepct],
pt.get_dataset('msmarco_passage').get_topics('dev.small'),
pt.get_dataset('msmarco_passage').get_qrels('dev.small'),
batch_size=200,
filter_by_qrels=True,
eval_metrics=['recip_rank'],
names=['bm25_terrier_stemmed', 'dph_terrier_stemmed', 'bm25_terrier_stemmed_text', 'bm25_bert_terrier_stemmed_text', 'bm25_terrier_stemmed_docT5query', 'bm25_terrier_stemmed_deepct'])