Duplicates in CRM and ERP: fix them for good instead of cleaning up every year.
Duplicates almost always come from the same places: an interface without a duplicate check, or a process where creating a new record is faster than finding the existing one. If you only clean up, they're back six months later. We find the source, close it and then clean up.
What are duplicates?
Duplicates are multiple records for the same customer, supplier or product, often with slightly different spellings, such as “Müller GmbH” and “Müller”. They distort reports, cause double mailings and wrong invoices, and slow down every automation, because no interface knows which record is the right one.
Duplicates don't come from laziness. They come from pragmatism.
At some point someone decided: creating a new record is faster than building logic. Thirty seconds instead of a sprint. Problem solved, at least for today.
Meanwhile the interface between shop system and ERP keeps running quietly. Every day, without a duplicate check. Because the check wasn't planned when it was set up, because nobody had the budget for it, and because it “works”.
That's how customer “Müller GmbH” appears three times. Then ten times. Then nobody knows which one is right.
Four ways duplicates get into your systems.
We find these sources almost every time, often several at once.
The interface without logic
Shop system and ERP exchange data, but nobody built a duplicate check when it was set up. Every order from an existing customer creates a new record, automatically and unnoticed.
The dummy customer
Customer number 12345 or “walk-in customer”. Thousands of transactions sit under one catch-all record because it was faster than proper assignment. For reporting and personalisation that's a total loss.
The unchecked migration
During the system switch, the old data was imported without cleanup because time was short. “We'll clean it up later” has become a foundation every new import piles onto.
Several systems, no shared truth
CRM, ERP, shop and accounting each keep their own customer list. The same company is “Müller GmbH”, “Müller” and “muellerGmbH”. No interface finds the match.
Clean up once or fix the cause?
| Cleanup project | Fix the cause |
|---|---|
| Starts by deleting duplicates | Starts by asking where they come from |
| Interfaces keep running as before | Interfaces get a duplicate check |
| Nobody is responsible for data quality | One owner for master data |
| Six months later: the next pile | Monitoring reports new duplicates right away |
How your data stays clean.
Not a years-long data governance project, but a concrete intervention that works right away.
Find the sources
Which interfaces, processes and dummy records produce duplicates?
Close the sources
A duplicate check in the interfaces and clear rules for creating new records.
Clean up
Merge duplicates using rules you can follow.
Monitor
New duplicates get reported before another pile builds up.
First the AI project, then the truth about the data.
An online retailer wants to use AI in customer service. The first look at the customer database shows: a large share of the records are duplicates, built up over years by an interface between shop and ERP without a duplicate check. Plus a dummy customer account with thousands of orders under it.
The AI project becomes a data project, but this time done properly. That's how we tackle cases like this as IT problem solvers.
- Duplicate sources found in the first week
- Interface retrofitted with duplicate logic
- Existing data cleaned up with traceable rules
- Monitoring reports new duplicates daily
- Dummy account dissolved, orders assigned
- Afterwards the AI runs on clean data
Cleaning up once is like weeding without pulling the roots.
What you usually ask us about duplicates.
Partly. Tools for automatic deduplication exist, but without prior analysis they merge wrong records as often as right ones. Only once it's clear which attributes make a customer unique, such as tax number, address or email, does automation pay off. Otherwise it makes the problem worse.
Not necessarily. Duplicates usually come from the interfaces that write data into the ERP, or from processes before that. That's where we start: a duplicate check in the interface and clear rules for creating new records. The ERP itself often stays untouched, and the risk to day-to-day operations stays small.
First visible results usually come after two to four weeks, depending on data volume and the number of systems. More important than speed is closing the sources at the same time. Otherwise everything starts over after a few months, and the cleanup was expensive and pointless.
In most companies nobody, and that's exactly the root of the problem. Data quality needs an owner who sets rules and spots deviations. We don't just help with the cleanup, but also with setting up clear responsibilities and simple processes that keep your data clean for good.
Your data grows. Your data quality doesn't.
Tell us briefly what it looks like for you. We'll tell you in a call where the duplicate sources are.