LLaRA
Free while signed in. Answers cite the passages they came from.
LLaRA adapts a decoder-only LLM for dense retrieval via two tailored pretext tasks that leverage text embeddings from the LLM itself.
EBAE pretext task: Embedding-Based Auto-Encoding uses LLM embeddings to reconstruct tokens of the input sentence, aligning the embedding space with semantic content.
EBAR pretext task: Embedding-Based Auto-Regression predicts tokens of the next sentence from the current embedding, injecting discourse-level signal into retrieval embeddings.
LLaMA 2 7B base: A LLaMA 2-7B base model is adapted into a retriever with these pretext tasks, yielding significant gains on MSMARCO and BEIR.
Decoder retrievers validated: Provides another data point that decoder-only LLMs, with the right adaptation, rival specialized encoder retrievers - a theme that continued through 2024.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack