We introduce REPLUG, a retrieval-augmented language modeling framework that
treats the language model (LM) as a black box and augments it with a tuneable
retrieval model. Unlike prior retrieval-augmented LMs that train language
models with special cross attention mechanisms to encode the retrieved text,
REPLUG simply prepends retrieved documents to the input for the frozen
black-box LM. This simple design can be easily applied to any existing
retrieval and language models. Furthermore, we show that the LM can be used to
supervise the retrieval model, which can then find documents that help the LM
make better predictions. Our experiments demonstrate that REPLUG with the tuned
retriever significantly improves the performance of GPT-3 (175B) on language
modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by
5.1%.
Description
REPLUG: Retrieval-Augmented Black-Box Language Models
%0 Generic
%1 shi2023replug
%A Shi, Weijia
%A Min, Sewon
%A Yasunaga, Michihiro
%A Seo, Minjoon
%A James, Rich
%A Lewis, Mike
%A Zettlemoyer, Luke
%A Yih, Wen-tau
%D 2023
%K llm machinelearning
%T REPLUG: Retrieval-Augmented Black-Box Language Models
%U http://arxiv.org/abs/2301.12652
%X We introduce REPLUG, a retrieval-augmented language modeling framework that
treats the language model (LM) as a black box and augments it with a tuneable
retrieval model. Unlike prior retrieval-augmented LMs that train language
models with special cross attention mechanisms to encode the retrieved text,
REPLUG simply prepends retrieved documents to the input for the frozen
black-box LM. This simple design can be easily applied to any existing
retrieval and language models. Furthermore, we show that the LM can be used to
supervise the retrieval model, which can then find documents that help the LM
make better predictions. Our experiments demonstrate that REPLUG with the tuned
retriever significantly improves the performance of GPT-3 (175B) on language
modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by
5.1%.
@misc{shi2023replug,
abstract = {We introduce REPLUG, a retrieval-augmented language modeling framework that
treats the language model (LM) as a black box and augments it with a tuneable
retrieval model. Unlike prior retrieval-augmented LMs that train language
models with special cross attention mechanisms to encode the retrieved text,
REPLUG simply prepends retrieved documents to the input for the frozen
black-box LM. This simple design can be easily applied to any existing
retrieval and language models. Furthermore, we show that the LM can be used to
supervise the retrieval model, which can then find documents that help the LM
make better predictions. Our experiments demonstrate that REPLUG with the tuned
retriever significantly improves the performance of GPT-3 (175B) on language
modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by
5.1%.},
added-at = {2023-03-02T05:31:59.000+0100},
author = {Shi, Weijia and Min, Sewon and Yasunaga, Michihiro and Seo, Minjoon and James, Rich and Lewis, Mike and Zettlemoyer, Luke and Yih, Wen-tau},
biburl = {https://www.bibsonomy.org/bibtex/2c2cfdb13f9ddd85c3ee12f88de85b14a/sairahul},
description = {REPLUG: Retrieval-Augmented Black-Box Language Models},
interhash = {2592a3b3900add5ea65ea5c97a0ae0e9},
intrahash = {c2cfdb13f9ddd85c3ee12f88de85b14a},
keywords = {llm machinelearning},
note = {cite arxiv:2301.12652},
timestamp = {2023-03-02T05:31:59.000+0100},
title = {REPLUG: Retrieval-Augmented Black-Box Language Models},
url = {http://arxiv.org/abs/2301.12652},
year = 2023
}