The Near-Miss Merge: A Defensible Deduplication Framework for UK Capture Teams
Deduplication is usually sold as housekeeping. A capture platform flags two records as a probable match, an operator accepts the suggestion, and the estate becomes marginally tidier. In practice, that small decision can be a records management decision with lasting consequences. Two invoices that look alike may represent separate payments. Two forms that differ by a tick box may record different instructions. Correspondence from two sites may be near-identical in wording but distinct in origin.
For UK capture teams in 2026, the risk is not that deduplication is unnecessary. The risk is that it is treated as a mechanical tidy-up rather than a controlled change to the record. A defensible routine begins with the assumption that a near-match is not automatically a duplicate.
Classify before you compare
Most deduplication failures start with a narrow question: do these records look the same? A safer starting point is to ask what kind of repetition is present. Capture teams should maintain a simple duplicate taxonomy and apply it before any matching rule is run.
- Exact copies. The same file, page set, or image ingested twice, often through a retry, a mailbox rule, or a manual upload. These are usually safe to suppress at ingestion, provided the original source and timestamp are retained.
- Re-scans. The same physical document scanned again because the first image was poor, misfiled, or captured at the wrong resolution. The later image may replace the earlier one, but the earlier image should remain linked as capture history rather than silently overwritten.
- Near-duplicates. Records with small differences in OCR text, page order, scan quality, or form completion. These require evidence-based comparison, not a similarity score alone.
- Legitimate repeats. Separate invoices for instalments, repeated standard letters, amended forms, and multi-site correspondence. These are not duplicates in any meaningful sense and must not be merged simply because the template is familiar.
- Versions. A draft, a signed copy, and a subsequently amended form may share most of their content. They are related records, not interchangeable ones.
This taxonomy does more than improve accuracy. It gives reviewers a shared vocabulary and makes the eventual decision easier to audit.
Use a matching hierarchy, not a single confidence score
A deduplication engine may return a percentage, but a percentage is not a rationale. UK capture teams should define a matching hierarchy that reflects the evidential weight of each field. Reference numbers and other hard identifiers sit at the top. Dates, supplier names, amounts, and content similarity sit below them.
A workable hierarchy has three tiers. The first tier uses exact identifiers: invoice number, purchase order reference, claim reference, case number, or a document control number. A match here is strong, but it still needs a sanity check against document type and date. The second tier uses composite evidence: supplier name plus invoice date plus amount, or site code plus form type plus period. No single field in this tier should trigger an automatic merge. The third tier uses content similarity and OCR text overlap. This tier should generate candidates for review, not decisions.
The hierarchy should be documented and versioned. If the matching rules change, the change should be recorded in the same way as any other capture control. That way, an auditor can see not only what was merged, but which rule set was in force at the time.
Put a human gate around ambiguity
Automatic suppression is appropriate for exact copies where the source, timestamp, and ingestion route are known. It is rarely appropriate for near-duplicates. Ambiguous cases should enter a review queue with a clear evidence pack: both records side by side, the fields that matched, the fields that differed, the source location, and the proposed action.
Review should be assigned to someone with the authority to make a records decision, not merely the ability to click a button. For smaller teams, that may be a senior capture operator. For larger teams, it may be a records manager or a compliance lead. A useful control is the two-role rule: one person proposes the merge or deletion, and another approves it when the record is subject to retention, legal hold, or financial audit.
The reviewer should be able to choose from a small set of outcomes: retain both records and link them as related; retain both records with no link; merge metadata while preserving both source objects; suppress one record from the working view while retaining it in the archive; or delete under an approved retention rule. The important point is that the reviewer is not forced into a binary choice between duplicate and not duplicate.
Record every decision, including the uneventful ones
An audit trail for deduplication should answer five questions: what was compared, what rule was applied, who decided, what action was taken, and where the original records now sit. The trail should include the record identifiers before and after the action, the match tier, the field-level evidence, the reviewer, the approver, the date, and any linked records.
Teams should also log the cases where no action was taken. A reviewed near-match that was deliberately retained as two records is useful evidence that the process is working. It shows that the team is not optimising for a lower duplicate count at the expense of record integrity.
Make the routine operational
Deduplication works best as a scheduled control rather than an occasional clean-up. A monthly review can examine new candidates, while a quarterly sample can test whether earlier merges remain defensible. Metrics should include the number of candidates reviewed, the proportion retained as separate records, the number of merges reversed, and the time taken to clear the review queue. A falling duplicate count is not, by itself, a sign of success.
In the UK context, deduplication also touches retention and data protection obligations. Merging records can change what is held, how it is linked, and how it is retrieved. The routine described here is not legal advice, but it supports the kind of documented, proportionate decision-making that UK organisations should be able to evidence. The safest principle is simple: when two records are close but not identical, close is not the same as duplicate. Treat the merge as a decision, not a tidy-up.