REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Abstract

Dense retrieval over long documents is expensive: token-level encoders scale quadratically in sequence length and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs and primarily document-to-document retrieval; single-chunk inputs are served by the GN alone. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets per regime: statistically indistinguishable from models 1.6–4.3× larger on the patent task, and within 0.65 nDCG@10 of a 20×-larger model on LoCo.

Method

REIGN architecture: a document is split into chunks, each chunk is encoded by a frozen Guidance Network, and a trainable REIGN encoder aggregates the chunk embeddings into a single document embedding by average pooling. — **REIGN architecture.** A long document is split into chunks and a frozen Guidance Network (GN) encodes each chunk once into a contextual embedding, which can be cached to disk and reused. The trainable REIGN encoder is a small Transformer over that chunk-embedding sequence: a permutation-equivariant set function with no positional encoding. Average pooling across chunks yields one document embedding, invariant to chunk order. Single-chunk inputs need no aggregation and are served by the GN alone.

Results

1.6–4.3×

Parity at a fraction of the parameters

On the DAPFAM patent task, REIGN + GTE-large (357M total) is statistically indistinguishable from dense baselines 1.6–4.3× larger, under paired significance tests on per-query nDCG@100.

49–229×

Long-context efficiency

With cached GN chunk embeddings, REIGN answers queries 49–229× faster than re-running the GN per query, and stays at parity when uncached. Peak GPU memory is 0.24–1.73 GB, versus 4.8–18.9 GB for the native long-context dense baselines (Jina-v3 measured at batch 4).

0.65

Close to a 20×-larger model on LoCo

On the out-of-distribution LoCo suite, REIGN + GTE-large is within 0.65 nDCG@10 of a 20×-larger model.

System	Params	nDCG@10	Δ
Sparse lexical
BM25	–	76.82	–
TF-IDF	–	74.19	–
Native long-context dense (truncate to 8K)
BGE-M3	568M	67.21	–
Jina-Embeddings-v3	572M	63.93	–
Stella-en-1.5B-v5	1.54B	63.66	–
Nomic-Embed-v1.5 ^‡	137M	11.05	–
Bare GN (truncated, 512)
GTE-small	33M	62.37	–
GTE-base	110M	63.48	–
GTE-large	335M	63.58	–
BGE-base	110M	61.95	–
BGE-large	335M	63.12	–
Bare GN (chunked mean-pool, 512)
GTE-small	33M	65.10	+2.73
GTE-base	110M	65.36	+1.88
GTE-large	335M	65.94	+2.36
BGE-base	110M	64.07	+2.12
BGE-large	335M	64.79	+1.67
REIGN (this work)
REIGN + GTE-small	55M	67.31	+4.94
REIGN + BGE-base	132M	65.17	+3.22
REIGN + GTE-base	132M	66.03	+2.55
REIGN + BGE-large	357M	66.45	+3.33
REIGN + GTE-large	357M	66.73	+3.15

GoodWiki-Long test. 5,854 queries against a 53,562-document corpus, top-10 retrieval; nDCG@10 in percent. Δ is the lift over the truncated baseline of the same GN. REIGN uses the base-l3 encoder (22M) at the best-performing stride per GN; best per column in bold. ^‡ Nomic-Embed-v1.5's native training context is 2K tokens; its 8K ceiling is RoPE-extended and marked experimental by its authors, and it collapses on documents whose median length already exceeds 5K words.

Three scatter plots of total parameters (log scale) against retrieval quality for GoodWiki-Long, DAPFAM, and LoCo, comparing REIGN against chunked and native long-context dense baselines. — **Parameter efficiency across three long-document retrieval regimes.** Total parameters (log scale) versus the primary nDCG metric; solid lines connect the three guidance-network scales (small / base / large), so the REIGN cross-chunk encoder is visible against its own chunked mean-pool baseline. The picture is task-conditional: REIGN sits above the chunked baseline at every scale on GoodWiki-Long, matches or exceeds it on DAPFAM, and on LoCo the bare chunked baseline is at or above REIGN except at the small-GN end.

Dataset

We release GoodWiki-Long-Synthetic, a synthetic long-document retrieval dataset built from GoodWiki, a cleaned markdown release of Wikipedia. Long articles are rephrased with an LLM, so each article and its rephrasal form an aligned positive pair; topical distractors are injected as partial matches. The result is graded relevance over documents that average 5,065 words. The dataset uses the standard BEIR/MTEB tri-config layout and works with existing retrieval evaluation pipelines: huggingface.co/datasets/devrim/goodwiki_long_synthetic_ir.

BibTeX

@inproceedings{cavusoglu2026reign,
  title         = {{REIGN}: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling},
  author        = {{\c{C}}avu{\c{s}}o{\u{g}}lu, Devrim and Akba{\c{s}}, Emre},
  booktitle     = {Findings of the Association for Computational Linguistics: {EMNLP} 2026},
  year          = {2026},
  publisher     = {Association for Computational Linguistics},
  eprint        = {2608.29899},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  note          = {To appear}
}