Entity resolution, also known as record linkage, deduplication, entity matching, or identity resolution, is the process of identifying and linking records that refer to the same real-world entity. Such an entity could be an individual, a business, or a product, and is often represented by quasi-identifying information like a name and other characteristics. In large databases, names do not uniquely identify entities due to duplication. Additionally, many entity attributes change over time, such as an individual moving and changing their address. Complicating matters further, databases often contain errors and variations. In this context, identifying unique entities becomes a complex challenge that necessitates intelligent algorithmic solutions.
Consider the following two product descriptions as an example. Are they the same product?
Product A | Product B |
Apple 2022 MacBook Pro Laptop with M2 chip: 13-inch Retina Display, 8GB RAM, 256GB SSD Storage | Apple MacBook Pro M2 Chip (13-inch, 8GB RAM, 256GB SSD Storage) - Space Gray (2022 Model) |
At first glance, it might seem like they are the same product. However, the color of Product A is not specified, and Product B does not mention having a Retina display. Depending on the ultimate goals of an entity resolution application, we may want to link these two products or keep them separate.
What about these two customer records?
Customer 1 | Customer 2 |
Name: John Doe Email: john.doe@email.com Address: 123 Main St, Springfield, IL Birthday: 01/15/1985 | Name: Johnathan Doe Email: johndoe1985@email.com Address: 123 Main Street, Springfield, IL Birthday: 15-01-1985 |
Despite differences in names, email addresses, and formatting differences for the address and birthday, these are likely the same person. An entity resolution system used to aggregate individual customer records would link them.
Large-scale entity resolution systems make billions of decisions like these to identify useful relationships between records. To do so, they use efficient data structures, string comparison functions, natural language processing, deterministic rules, and classification and clustering algorithms.
By understanding and implementing entity resolution, businesses and organizations can better manage their data, reduce redundancy, and enhance the overall quality of their databases.
What About Valires?
At Valires, we are entity resolution experts focusing on helping businesses and cloud entity resolution providers validate their entity resolution systems. We are on a mission to address the performance and trust issues that plague too many entity resolution systems. We believe that entity resolution can play a crucial role to in addressing data quality, data integration, and relationship discovery problems, and we provide the evaluation pipeline necessary to achieve this goal.