DATA DEDUPLICATION METHOD AND APPARATUS， COMPUTER DEVICE， AND STORAGE MEDIUM
收藏国家林业和草原科学数据中心2023-02-12 更新2024-03-07 收录
下载链接：
https://www.forestdata.cn/dataDetail.html?id=38a62cc1-c9a0-4b0d-8215-0094b44e6d15
下载链接
链接失效反馈官方服务：
资源简介：
Disclosed are a data deduplication method and apparatus， a computer device， and a storage medium. The method comprises: obtaining a data access request and extracting feature fields therein; cleaning the feature fields and performing normalization processing on the cleaned feature fields; connecting the feature fields to generate a feature field combination， and compressing the feature field combination by using a Hash algorithm; identifying the compressed feature fields on the basis of a preset database cluster， and determining， according to the identification result， whether the feature field is a duplicate field; if the feature field is a duplicate field， storing the feature field to a preset exception handling queue; if the feature field is not a duplicate field， outputting a prompt message used for prompting that the feature field is a normal field.
提供机构：
国家林业和草原科学数据中心
创建时间：
2023-02-12
