Entity Resolution (ER) involves identifying, linking, and differentiating distinct entities within complex datasets. The use of deep embedding models has significantly advanced ER by enabling the use of varied data types including free-form text, images, and more. However, despite these advancements, the added complexity of embedding models can cause unexpected failures, warranting comprehensive testing and evaluation procedures.
Deep Embedding Models: Processing Multimodal Data
Deep embedding models (e.g., word embeddings), a type of machine learning algorithms, can embed high-dimensional, multimodal data into a transformed space that exposes relevant semantic features. Particularly, these models can be used as a first step to assess the similarity of complex data like free-form text or images. Using an appropriate embedding model, computing topic similarity between two sentences can be as simple as computing the cosine similarity between the embeddings. For more fine-grained control, the embeddings can be used as input features of another neural network or machine learning algorithm.
For example, PatentsView uses sentence embeddings to compare patent topic similarity. This is one of the step involved in the disambiguation of unique inventors within the USPTO's patent database. Other research addresses the use and tuning of embeddings for ER tasks (see also here).
While these capabilities allow for a sophisticated comparison of different data attributes, they also introduce a higher degree of complexity into ER systems. As a result, robust testing and evaluation methodologies are essential to ensure the effectiveness of these ER systems and identify potential failure modes.
Testing and Evaluation: Ensuring ER System Effectiveness
Robust testing and evaluation procedures play a critical role in optimizing ER systems. They provide a detailed analysis of ER system performance, helping identify successful models and uncover performance issues.
At Valires, we consider three aspects to evaluation:
Estimation. Accurately estimate key performance metrics, even when benchmark datasets may be small or biased.
Understanding. Understand key characteristics of any ER system, analyze errors, and areas of underperformance.
Comparison. Compare the performance of competing systems and know which one is best.
All three aspects can be covered using the entity-centric evaluation methodology that we advocate for. This methodology relies on a set of disambiguated entities used as the "ground truth" or as a point of reference. This data can be found from external sources or obtained through manual effort. While it is not always easy to obtain correctly disambiguated entities, we have developed guidelines and methodologies to do so.
Given a set of disambiguated entities, you can then immediately use the ER-Evaluation Python package to produce performance estimates, summary statistics, comparisons, and visualizations. The package is built to be user-friendly. Once you have the required data, you can call simple functions that produce relevant statistics and interactive visualizations.
Assessing Deep Embedding Models for ER
Considering the end result of an ER system allows you to evaluate the impact of any of its internal component, including the choice of deep embedding models. Using our evaluation methodology, you can simply test different models and see the impact on reliable performance estimates. There is no need to carry out a detailed manual review for each model being tested. The same benchmark dataset and code can be used to assess any model and to compare models across them.
The Power of Collaboration
Our testing and evaluation methodologies for ER have emerged from collaborations, and are rooted in open-source software and published methodologies. This openness facilitates the sharing of feedback and direct contributions to open-source tools. We hope to see the advance of evaluation standards help drive methodological research in ER and its applications. The use of deep embeddings for ER is one of the many areas where many developments are yet to come.