REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Abstract

Dense retrieval over long documents is expensive: token-level encoders scale quadratically in sequence length and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs and primarily document-to-document retrieval; single-chunk inputs are served by the GN alone. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets per regime: statistically indistinguishable from models 1.6–4.3× larger on the patent task, and within 0.65 nDCG@10 of a 20×-larger model on LoCo.

Method

REIGN architecture: a document is split into chunks, each chunk is encoded by a frozen Guidance Network, and a trainable REIGN encoder aggregates the chunk embeddings into a single document embedding by average pooling. — **REIGN architecture.** A long document is split into chunks and a frozen Guidance Network (GN) encodes each chunk once into a contextual embedding, which can be cached to disk and reused. The trainable REIGN encoder is a small Transformer over that chunk-embedding sequence — a permutation-equivariant set function with no positional encoding — and average pooling across chunks yields a single document embedding that is invariant to chunk order. Single-chunk inputs need no aggregation and are served by the GN alone.

Results

1.6–4.3×

Parity at a fraction of the parameters

On the DAPFAM patent task, REIGN + GTE-large (357M total) is statistically indistinguishable from dense baselines 1.6–4.3× larger, under paired significance tests on per-query nDCG@100.

49–229×

Long-context efficiency

With cached GN chunk embeddings, REIGN answers queries 49–229× faster than re-running the guidance network per query, and stays at parity when uncached. Peak GPU memory is 0.24–1.73 GB, versus 4.8–18.9 GB for the native long-context dense baselines (Jina-v3 measured at batch 4).

0.65

Close to a 20×-larger model on LoCo

On the out-of-distribution LoCo suite, REIGN + GTE-large lands within 0.65 nDCG@10 of a 20×-larger model.

Three scatter plots of total parameters (log scale) against retrieval quality for GoodWiki-Long, DAPFAM, and LoCo, comparing REIGN against chunked and native long-context dense baselines. — **Parameter efficiency across three long-document retrieval regimes.** Total parameters (log scale) versus the primary nDCG metric; solid lines connect the three guidance-network scales (small / base / large), so the REIGN cross-chunk encoder is visible against its own chunked mean-pool baseline. The picture is task-conditional: REIGN sits above the chunked baseline at every scale on GoodWiki-Long, matches or exceeds it on DAPFAM, and on LoCo the bare chunked baseline is at or above REIGN except at the small-GN end.

Dataset

We release GoodWiki-Long-Synthetic, a synthetic long-document retrieval dataset built from GoodWiki, a cleaned markdown release of Wikipedia. Long articles are rephrased with an LLM so that each document and its rephrasal form a semantically aligned positive pair, and semantically overlapping topical distractors are injected as partial matches, giving a graded relevance structure over documents averaging 5,065 words. It ships in the canonical BEIR/MTEB tri-config layout, so it drops into standard retrieval evaluation pipelines: huggingface.co/datasets/devrim/goodwiki_long_synthetic_ir.

BibTeX

@inproceedings{cavusoglu2026reign,
  title     = {{REIGN}: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling},
  author    = {{\c{C}}avu{\c{s}}o{\u{g}}lu, Devrim and Akba{\c{s}}, Emre},
  booktitle = {Findings of the Association for Computational Linguistics: {EMNLP} 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  note      = {To appear}
}