lgoyal6 / noisy-rag-bench

noisy‑rag‑bench.

Retrieval benchmarks run on clean text. Documents inside a bank or a law firm are scans: OCR confusions, running headers, hyphens across line breaks, redacted spans. Nobody publishes what that does to a RAG pipeline, and it is the normal input for anyone deploying on premise.

Retrieval quality and answer quality come apart under noise, and the gap is invisible to the metric most systems are watched on. Drag the noise below and watch the two separate.

recall at 5% OCR
0.98 unchanged
answers recovered
0.51 from 1.00
after an 11-line fix
0.83 no new model
whole stack
261 MB 3.1 ms p50, CPU only
Try it

Wreck a document and look for the answer in it

Put a passage in the box and pick what kind of scan it came off. The benchmark's own noise functions run in your tab and do the damage. Then say what answer you would be looking for, and see whether it survived.

starting the engine noise.py and retrieval.py, unmodified
Noise
Amount 20%
character error rate
–
answer in the clean text
–
answer after the noise
–
after canonicalizing
–

The thing to notice

It opens on OCR at 20%, which reads as a 5% character error rate and has taken the answer away. Now switch to running headers and push it to 50%: the same 5% character error rate, and the answer is untouched. Two documents a cleaning budget would rank identically, and only one of them is broken.

Then watch the last column. Canonicalizing before indexing puts most of it back, which is eleven lines and no new model, and it is the entire recommendation of this repository.

looks small
The character error rate.Header noise scores 25.9% and costs nothing.
is fatal
Damage inside the answer span.OCR at 5% halves what is recoverable.
is cheap to fix
Canonicalize before indexing.The last column, eleven lines.
Figure 1

The dashboard says the system is fine

Each step adds document noise of one kind. The solid line is what a retrieval dashboard shows you. The dashed line is whether the answer was still in there.

132 questions over 132 chunks MiniLM in pure NumPy, no PyTorch, no network
Noise
Retriever
Amount
condition
-
character error
-
recall@5
-
answer recovered
-
after canonicalizing
-

What the two lines mean

recall@5 asks whether the right chunk came back at all. It is the number a retrieval dashboard plots, and under OCR noise it barely moves. Answer recovered asks whether the string that answers the question is still legible inside that chunk.

Solid and dashed rather than two colours, so the separation survives being printed or read by someone who does not see colour.

if you watch one metric
Watch the wrong one and you see nothing.Recall 0.98 while answers hit 0.51.
the cheap fix
Canonicalize before indexing.0.51 to 0.83, eleven lines, no new model.
Figure 2

How much noise matters less than which kind

Character error rate is a poor predictor of damage. Sorted by how badly each condition hurt the answer, not by how noisy it looks.

Page header contamination produces a higher character error rate than the five percent OCR case and costs nothing on either metric, because the added text is uniform and the answer span is untouched.

So a cleaning budget aimed at the noisiest-looking documents is aimed in the wrong place.

worst for answers
scan-degraded0.38 recovered, 36.9% character error.
free, despite looking worse
header contamination1.00 recovered, 25.9% character error.
Figure 3

Where it loses

Both losing axes, published, because a benchmark that only reports where its author wins is an advertisement.