The safe way to clean up duplicate records is to first define a matching rule that finds records that are the same, decide which record stays in each group, and never delete anything without approval. A full backup is taken before the cleanup starts and every merge is written to a report, so a record that was merged by mistake can be restored.
When a customer list or product catalogue is used for years, the same record gets entered again by different people at different times. "ABC Ltd.", "A.B.C. Limited", "Abc Ltd" and "ABC LIMITED COMPANY" are the same company, but to the system they are four separate customers. As a result, reports count the same customer more than once, the sales history is split into pieces, and two separate quotes go to the same company. In this article we explain how such a list is cleaned step by step, in a way that can be undone.
Why do duplicate records appear?
Most of the causes look innocent one by one. Nobody searches before entering a record, or the search fails to find the old record because of a spelling difference. One person uses an abbreviation, another types the full company name. Special characters are sometimes typed and sometimes not. Lists from two different systems are combined and nobody matches them. During a bulk import from Excel, the same file is uploaded twice. Over the years these small differences add up and the list looks far more crowded than it really is.
Step 1: Define the matching rule
The basis for deciding that two records are the same should be written down at the start. The most reliable criterion is an identifier that does not change: a tax number, a product code or an email address. If this field is filled in on every record, matching largely solves itself.
If there is no such field, or it is incomplete, you have to look at name similarity. To do this, spelling differences are removed first: upper and lower case, punctuation, spaces and suffixes such as "Ltd.", "Inc." or "LLC" are brought to the same form before comparing. Similar names are then confirmed with a second piece of information such as phone, city or address. Relying on name similarity alone is risky: two different companies or two different people can carry the same name.
Step 2: Decide which record stays in each group
Records found to be the same form a group. In each group one record stays as the "master record" and the others are merged into it. A simple rule is enough to choose which one stays: the record with the most transactions linked to it, the most recently updated record or the most complete record. A phone number or address missing from the master record can be filled in from the others.
The point to watch here is that the history linked to a duplicate is not lost. If a customer's old orders, invoices or correspondence are linked to the record that will be deleted, they must first be moved to the master record. Otherwise the cleanup simplifies the list but also wipes out the history.
Step 3: Get approval before deleting
However well the matching is set up, some records remain doubtful. That is why the cleanup is done in two stages. First, the tool or a person prepares a proposal list showing which records will be merged. Certain matches can be approved in bulk; doubtful ones are reviewed one by one by the person who knows the list best. The merge is carried out after approval.
This step may seem to take time, but it is what actually saves time. Instead of examining thousands of records one by one, only the few hundred that cannot be decided with certainty are checked.
Step 4: Prepare a way back
A full backup of the list should be taken before the cleanup begins. For every merge, which record was joined to which and which field changed should be written to a report. That way, if a few weeks later someone says "these two companies were actually different", only that merge can be reversed; there is no need to redo the whole cleanup.
We describe the tool that tidies up records built up over the years in this way on the duplicate record cleanup page: it finds similar records, suggests which one should stay, merges them in bulk after approval and reports everything that changed in a single report.
How do you keep the list tidy after the cleanup?
A list that has been cleaned once becomes messy again within a few months if no entry rule is set. A warning that shows whether a similar record already exists when a new one is entered, mandatory fields and a single spelling rule largely prevent this. If bulk imports are done, they should be checked with the same matching rule before uploading; we covered this in the article bulk-entering an Excel list into a system.
Things to watch out for
Cleaning up duplicate records is often the first step of an automation project; a tool built on messy data multiplies the errors faster. So the cleanup should be planned as a separate job and these questions answered at the start: Which fields will we look at to decide that two records are the same? Which record stays in each group? Who approves doubtful matches? Where is the backup kept, and how is a merge reversed? We explained separately why data stands in the way of automation in the article why automation projects fail.
Frequently asked questions
How are duplicate records found?
First, look for a field that does not change, such as a tax number, product code or email address. If there is none, names are compared after removing case differences, punctuation and company suffixes, and similar records are confirmed with a second piece of information such as phone, city or address.
Should duplicate records be deleted straight away?
No. First a proposal list showing which records will be merged is prepared, doubtful matches are checked and the history linked to the record being deleted is moved to the master record. A full backup is taken before the cleanup.
Can a record that was merged by mistake be restored?
If every merge is written to a report together with which record was joined to which, that single merge can be reversed. That is why a backup and a change report should be prepared before the cleanup starts.
Is there a job that keeps repeating at your company? Tell us about it and we will build the tool that does it.
Tell us what you needOther articles
Quote, proforma and invoice: which one is prepared when?
A quote tells the customer the price and terms; a proforma is a preliminary invoice prepared before the sale is final, for payment or advance approval, and it is not a tax document; an invoice is the official document issued after the sale takes place. Since all three carry the same customer and product information, producing them from a single source prevents the errors that come from typing the same information three times by hand.
30 September 2026Read →Why do automation projects fail?
Automation projects usually fail not because the software is broken, but because the organisation around the work was never set up. The five most common reasons are: the scope growing along the way, messy data, only one person using the tool, no training for the team, and nobody owning maintenance after setup. All five can be discussed before the work begins.
29 September 2026Read →The AI assistant on your website and KVKK: where are chats stored, and for how long?
When you put an AI assistant on your website, the messages visitors write may be personal data. In the URİKEM AI Customer Assistant, chats are stored on a server in Istanbul, are deleted automatically 180 days after the last message, and the business can delete a chat at any time. Messages are sent to an AI provider abroad so that a reply can be generated; visitors are informed of this and asked for consent before their first message.
29 September 2026Read →What should you check before having a website built?
When having a website built, most attention goes to design, yet the issues that cause the most trouble later stay on the unseen side: whose name the domain and hosting are registered under, who holds the text, how fast the site opens on a phone, and whether it has been submitted to Google. A company that settles these four points from the start also removes the risk of losing its site years later.
28 September 2026Read →Does automating social media make a brand feel like a robot?
Automating social media does not make a brand feel robotic; what does is text published without anyone reading it and comments left unanswered. In the right setup, the calendar, preparation and timed publishing run automatically; the brand's voice, final approval and conversation with customers stay with a person.
28 September 2026Read →Is it safe to move our data to the cloud, or should we keep it on our own server?
The cloud is neither safe nor unsafe in itself; what determines security is whose responsibility the data sits under, who can access it and how it is backed up. If you can answer these three questions clearly, your data is safe whether it is on your own server or in the cloud; if you cannot, changing its location will not solve the problem.
27 September 2026Read →What determines the cost of AI software?
The cost of AI software is made up of four items: setting up the work, monthly use and hosting, the usage fee for the AI model, and maintenance. In a ready-made package these items are combined into a single monthly fee; for a custom tool, a separate quote is given according to the size of the job. The hidden cost is often in maintenance and data preparation.
27 September 2026Read →Do you have to prepare the monthly sales report by hand every month?
If the report is prepared every month from the same source, in the same layout and with the same calculations, you do not have to prepare it by hand. An automation that takes the data and produces the tables, charts and summary by the same rules reduces copy-paste errors and delivers the report in the same format every month.
26 September 2026Read →What does an unanswered phone cost a business?
The cost of an unanswered phone is the customer who made that call ringing another company instead. An online booking system lets customers book their own appointment outside business hours and at busy times too; reminders and easy cancellation reduce the slots that go unused.
26 September 2026Read →Which questions can an AI customer assistant answer?
An assistant answers well the questions that can be answered with the information the company gives it: prices, opening hours, address, scope of services, appointments and return terms. Questions that require a decision, negotiation or individual review should be passed to a person without guesswork.
25 September 2026Read →Turning hundreds of PDFs and documents into one table: by hand or with AI?
If there are few documents and they share the same layout, manual entry may be enough. If documents number in the hundreds, their layouts differ and new ones keep arriving, reading them with AI and turning them into a table makes more sense; but suspicious fields must be checked by a person.
25 September 2026Read →How many days does an automation project take?
Duration is set not by how much code is written but by how organized the data is and who needs to approve, and when. Even a simple task can stretch out as more approvers get involved.
23 September 2026Read →Does a small company need an AI budget?
The budget is not set by the size of the company but by how many hours a specific task wastes. A small company can also start with a single repeating task and a measurable budget.
23 September 2026Read →Who benefits from an appointment system?
An appointment system can help when customers need to reserve a time slot and arranging those slots by phone interrupts your work. First clarify service durations, shared resources and exceptions.
22 September 2026Read →What if AI makes things up?
A fluent answer is not verified information. A business tool needs more than well-written output: it needs a workflow that prevents actions based on incorrect or incomplete information.
22 September 2026Read →Can a spreadsheet with thousands of rows be entered into a system in bulk?
In most cases, yes. The hard part is not the entry itself but getting the data into the shape the system expects and verifying that every row was really saved. If the system offers no bulk route, the work goes through the screen, and that depends on the rules of that system.
21 September 2026Read →Why does preparing a quote take so long?
Writing the quote does not take long; the time goes into hunting for the price, copying and correcting an old quote, and rebuilding the layout every time. Once the price list sits in one place and the rules are written down, a quote comes out in minutes.
21 September 2026Read →Which jobs are suited to automation, and which are not?
The job best suited to automation is one that repeats often, whose rules can be written down and whose data is already digital. Work that needs judgement, is full of exceptions or has its data on paper needs another step first.
18 September 2026Read →What if the AI assistant on your site gives a wrong answer?
The risk is real, but manageable. An assistant's wrong answer usually comes from guessing about something it does not know. The solution is to draw a clear line from the start around what the assistant knows and does not know, and to have it hand over to a person when it does not know.
18 September 2026Read →Is ranking in Google the same as being mentioned by AI?
Partly. Both draw on the same foundation, but they do not reward the same thing: Google ranks a page, while AI reads several sources, writes an answer and decides for itself who to mention in it.
17 September 2026Read →Why doesn't ChatGPT mention my company?
The reason is rarely “the AI does not like us”. Usually it is a technical block, scattered company information, or the company not being mentioned anywhere else online. All three can be fixed.
17 September 2026Read →What should your site have so AI describes you correctly?
When AI describes a company, it uses the clearest sentences it finds on the site. The list below goes page by page through where those sentences should sit, how they should be written and the markup machines will read.
17 September 2026Read →What is GEO? Being visible in AI answers
GEO (Generative Engine Optimization) is the work of making sure that when ChatGPT, Gemini, Perplexity and Google's AI Overviews answer a question, they find your business, describe it correctly and cite it as a source. It does not replace SEO; it builds on top of it.
17 September 2026Read →Will an AI tool learn my company's data?
Short answer: not in a tool running through a commercial API. But that answer depends on how the setup was done, so it is worth knowing what goes where.
16 September 2026Read →When does a spreadsheet stop being enough?
Excel is not a bad tool; it handles most jobs and more. The problem is carrying on with it for months after the point where it stopped being the right fit.
16 September 2026Read →Off-the-shelf software or custom-built?
The right answer depends on how specific the work is to you. Work that everyone does the same way is cheaper off the shelf; work you do your own way is cheaper built for you.
16 September 2026Read →