Duplicate Detection with Generative AI

Welcome to learning edition of the Data Pragmatist, your dose of all things data science and AI.

📖 Estimated Reading Time: 5 minutes. Missed our previous editions?

• How to Measure Data Quality in Data Environments

• The Rise and Fall of Early AI Gadgets

🧠 Duplicate Detection with Generative AI

Customer data in CRM systems often suffers from replication issues, leading to complex downstream processes. Traditional NLP-based Entity Matching (EM) techniques address this by comparing records pairwise, but advancements in Large Language Models (LLMs) and Generative AI can significantly enhance duplicate detection and resolution. My research on benchmark datasets showed an improvement in de-duplication accuracy from 30% with traditional NLP to nearly 60% using my proposed method.

Traditional Approach

• Data Preparation: This step involves cleaning data by removing non-ASCII characters, capitalizing, and tokenizing text to prepare it for NLP algorithms.

• Candidate Generation: Candidate records are generated by combining all records, excluding self-comparisons. To reduce the number of candidates, blocking is employed.

• Blocking: Blocking eliminates records that can't be duplicates due to different values in specific columns, such as city names in customer records.

• Matching: Candidate records are compared using traditional NLP similarity metrics to identify potential matches.

• Clustering: Matched records are grouped into clusters for easier identification.

Proposed Method

1. Create Match Sentences: Attributes of interest are concatenated into a single sentence, e.g., "John Hartley Smith 20 Main Street London."

2. Create Embedding Vectors: Match sentences are encoded into vector space using embedding models like all-mpnet-base-v2, resulting in 768-dimensional vectors.

3. Clustering: Embedding vectors are clustered using DBSCAN, utilizing distance metrics like L2 Norm and Cosine Similarity with an epsilon threshold. Clusters are formed by grouping records within the epsilon distance and sharing the same blocked column value.

Experiments and Results

Testing the proposed method on customer data and the Musicbrainz 200K benchmark dataset showed superior performance. Visualizations of clustering using UMAP reduction confirmed the effectiveness of this approach.

Top Data Influencers to Follow in 2024

1. Barr Moses

   • Role: CEO and co-founder of Monte Carlo

   • Specialty: Data reliability and data observability

   • Notable Contributions: Co-author of "Data Quality Fundamentals", coined the term “data downtime”

   • Platforms: Twitter

   • Predictions for 2023: Available here

2. Ben Rogojan

   • Alias: Seattle Data Guy

   • Role: Data engineering solutions architect, consultant

   • Specialty: Data architecture and statistics

   • Background: Ex-Meta data engineer

   • Platforms: LinkedIn (44K followers), YouTube (50K subscribers), Medium, Twitter

   • Content: Tips on data engineering, industry trends

3. Benn Stancil

   • Role: Co-founder and Chief Analytics Officer at Mode

   • Specialty: Collaborative business intelligence and interactive data science

   • Background: Ex-Microsoft senior analyst

   • Platforms: Substack, Twitter

   • Contributions: Industry discussions, particularly on the metrics layer in the modern data stack

4. Bruno Aziza

   • Role: Head of Data and Analytics at Google Cloud

   • Specialty: Big data analytics, leadership in tech

   • Background: Leadership roles at Microsoft, Oracle, and Google

   • Platforms: YouTube, Medium, Twitter

   • Content: AI, data, and analytics news and trends

5. Cassie Kozyrkov

   • Role: Chief Data Scientist at Google

   • Specialty: Data science, decision intelligence

   • Contributions: Founder of Decision Intelligence Engineering at Google, trained over 20K Googlers

   • Platforms: Widely recognized as a top advisor for AI and data-driven initiatives