Entity Resolution

🚀 Entity Recognition Tool

This tool offers a comprehensive approach to compare two sets of data, evaluating both the exact and semantic equality of their values. It leverages text normalization and language models to determine if the data comes from the same entity.

🚀 Quick Start

This entity recognition tool provides a comprehensive way to compare two sets of data, assessing the exact and semantic equality of their values. It uses text normalization and language models to determine if the data comes from the same entity.

✨ Features

• Text Normalization: Converts text to lowercase, removes punctuation, and standardizes whitespace.
• Numerical Comparison: Performs exact and semantic comparisons of numerical values (ignores order for lists).
• JSON Traversal: Iterates through each JSON object key by key and compares corresponding values using the compare_values function.
• Language Model Integration: Utilizes a generative language model to evaluate semantic similarity and ultimately determine if the data comes from the same entity.

📦 Installation

To use this tool, make sure you have installed the necessary dependencies. You can install them via pip:

pip install genai

💻 Usage Examples

Function Introduction

1. normalize_text(text):
   • Normalizes the input text, including converting to lowercase, removing punctuation, and standardizing whitespace.
2. compare_values(val1, val2):
   • Compares two values both exactly and semantically.
   • Ignores the order of elements when semantically comparing lists.
3. compare_json(json1, json2):
   • Compares each key of two JSON objects.
   • Uses the compare_values function to evaluate the values of each key.
   • Integrates a generative language model to evaluate semantic similarity and provides a final judgment.

Basic Usage

import json
import genai
import re

# Define your JSON objects
json1 = {
    "name": "John Doe",
    "address": "123 Main St, Anytown, USA",
    "hobbies": ["reading", "hiking", "coding"]
}

json2 = {
    "name": "john doe",
    "address": "123 Main Street, Anytown, USA",
    "hobbies": ["coding", "hiking", "reading"]
}

# Compare JSON objects
comparison_results = compare_json(json1, json2)

# Generate the final matching result
model1 = genai.GenerativeModel("gemini-2.0-flash-thinking-exp")
result_matching = model1.generate_content("综合这些信息，你认为可以判断两个数据来自同一主体吗？"+json.dumps(comparison_results, ensure_ascii=False, indent=4))
print(result_matching.text)

🤝 Contribution Guidelines

Any contributions are welcome! Please open an issue or submit a pull request.

📄 License

This project is licensed under the MIT License. For more details, please refer to the LICENSE file.

📞 Contact

If you have any questions or suggestions, feel free to contact me:

• Email: u3588064@connect.hku.hk
• GitHub: u3588064@connect.hku.hk

WeChat