Dataset for paper Spotting the Hook: Leveraging Domain Data for Advanced Phishing Detection

The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS certificate fields, and GeoIP information for 432,572 benign domains from Cisco Umbrella and 68,353 phishing domains from PhishTank and OpenPhish services. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered as phishing by VT have been removed. The data was collected between March and November 2023.The final assessment of the data was conducted in December 2023. The dataset is useful for statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing detection. Data Files The data is located in two individual files: benign.json - data for 432,572 benign domains, and phishing.json - data for 68,353 phishing domains. Data Structure Both files contain a JSON array of records generated using mongoexport. The following table documents the structure of a record. Please note that: some fields may be missing (they should be interpreted as nulls), extra fields may be present (they should be ignored), due to a processing error, the common_name field of the certificate objects always contains trailing symbols: ‘> . Field name Field type Nullable Description domain_name String No The evaluated domain name url String No The source URL for the domain name evaluated_on Date No Date of last collection attempt source String No An identifier of the source sourced_on Date No Date of ingestion of the domain name dns Object Yes Data from DNS scan rdap Object Yes Data from RDAP or WHOIS tls Object Yes Data from TLS handshake ip_data Array of Objects Yes Array of data objects capturing the IP addresses related to the domain name DNS data (dns field) A Array of Strings No Array of IPv4 addresses AAAA Array of Strings No Array of IPv6 addresses TXT Array of Strings No Array of raw TXT values CNAME Object No The CNAME target and related IPs MX Array of Objects No Array of objects with the MX target hostname, priority and related IPs NS Array of Objects No Array of objects with the NS target hostname and related IPs SOA Object No All the SOA fields, present if found at the target domain name zone_SOA Object No The SOA fields of the target’s zone (closest point of delegation), present if found and not a record in the target domain directly dnssec Object No Flags describing the DNSSEC validation result for each record type ttls Object No The TTL values for each record type remarks Object No The zone domain name and DNSSEC flags RDAP data (rdap field) copyright_notice String No RDAP/WHOIS data usage copyright notice dnssec Bool No DNSSEC presence flag entitites Object No An object with various arrays representing the found related entity types (e.g. abuse, admin, registrant). The arrays contain objects describing the individual entities. expiration_date Date Yes The current date of expiration handle String No RDAP handle last_changed_date Date Yes The date when the domain was last changed name String No The target domain name for which the data in this object are stored nameservers Array of Strings No Nameserver hostnames provided by RDAP or WHOIS registration_date Date Yes First registration date status Array of Strings No The state of the registered object [TODO] terms_of_service_url String No URL of the RDAP usage ToS url String No URL of the RDAP entity whois_server String No WHOIS server address TLS data (tls field) cipher String No TLS cipher suite description according to [TODO] protocol String No One of “TLS”, ”TLSv1.2”, ”TLSv1.3” certificates Array of Objects No Array of objects representing the certificate chain, the first element is the root certificate IP data (elements in the ip_data array) ip String No The IP address from_record String No The type of the DNS record the address was captured from remarks Object No Ping round-trip time, “is alive” flag and rdap/geo/asn evaluation dates rdap Object Yes RDAP data, similar to DNS RDAP, see the JSON Schema for details geo Object Yes Geolocation data from the GeoLite2 City database (e.g. latitude, longitude, city, country, etc.) asn Object Yes Autonomous system data from the GeoLite2 ASN database (ASN, organization, network) Acknowledgements We would like to thank the OpenPhish Team for grating permission to use and publish their dataset. We also thank VirusTotal for providing us access to the API for research purposes. The research has been supported by the Flow-based Encrypted Traffic Analysis project, no. VJ02010024, granted by the Ministry of the Interior of the Czech Republic and the Smart Information Technology for a Resilient Society project, no. FIT-S-23-8209, granted by Brno University of Technology.本数据集涵盖432,572个来自Cisco Umbrella的良性域名，以及68,353个来自PhishTank与OpenPhish服务的钓鱼域名，包含域名系统（DNS）记录、IP相关特征、WHOIS/RDAP信息、传输层安全（TLS）证书字段数据以及GeoIP地理信息。钓鱼数据集的真实标签已通过VirusTotal (VT)服务完成双重校验，被VT判定为非钓鱼的域名已被移除。数据采集周期为2023年3月至11月，数据集最终评估工作于2023年12月完成。 本数据集可用于域名数据统计分析，或用于提取特征以训练基于机器学习的分类器，例如钓鱼检测分类器。 数据文件 数据集存储于两个独立文件中： - benign.json：包含432,572个良性域名的数据集； - phishing.json：包含68,353个钓鱼域名的数据集。 数据结构 两个文件均为由mongoexport生成的JSON数组格式记录。下表说明单条记录的结构。请注意： 1. 部分字段可能缺失（此类字段应视为null值）； 2. 可能存在额外字段（此类字段应予以忽略）； 3. 因处理误差，证书对象的common_name字段始终带有尾部符号：‘> . ### DNS数据（dns字段） 以下为dns字段包含的子字段： | 字段名 | 字段类型 | 是否可为空 | 描述 | | --- | --- | --- | --- | | A | 字符串数组 | 否 | IPv4地址数组 | | AAAA | 字符串数组 | 否 | IPv6地址数组 | | TXT | 字符串数组 | 否 | 原始TXT记录值数组 | | CNAME | 对象 | 否 | CNAME目标及相关IP | | MX | 对象数组 | 否 | 包含MX目标主机名、优先级及相关IP的对象数组 | | NS | 对象数组 | 否 | 包含NS目标主机名及相关IP的对象数组 | | SOA | 对象 | 否 | 目标域名下找到的所有SOA字段 | | zone_SOA | 对象 | 否 | 目标域的父区域（最近的委托点）的SOA字段，仅当目标域名直接未找到该记录时存在 | | dnssec | 对象 | 否 | 各记录类型的DNSSEC验证结果标志 | | ttls | 对象 | 否 | 各记录类型的TTL值 | | remarks | 对象 | 否 | 区域域名及DNSSEC标志 | ### RDAP数据（rdap字段） 以下为rdap字段包含的子字段： | 字段名 | 字段类型 | 是否可为空 | 描述 | | --- | --- | --- | --- | | copyright_notice | 字符串 | 否 | RDAP/WHOIS数据使用版权声明 | | dnssec | 布尔值 | 否 | DNSSEC存在标志 | | entities | 对象 | 否 | 包含各类相关实体类型数组的对象（例如滥用、管理、注册人），数组内包含描述各实体的对象 | | expiration_date | 日期 | 是 | 当前过期日期 | | handle | 字符串 | 否 | RDAP句柄 | | last_changed_date | 日期 | 是 | 域名最后修改日期 | | name | 字符串 | 否 | 本对象中数据对应的目标域名 | | nameservers | 字符串数组 | 否 | RDAP或WHOIS提供的域名服务器主机名 | | registration_date | 日期 | 是 | 首次注册日期 | | status | 字符串数组 | 否 | 注册对象的状态[TODO] | | terms_of_service_url | 字符串 | 否 | RDAP使用服务条款的URL | | url | 字符串 | 否 | RDAP实体的URL | | whois_server | 字符串 | 否 | WHOIS服务器地址 | ### TLS数据（tls字段） 以下为tls字段包含的子字段： | 字段名 | 字段类型 | 是否可为空 | 描述 | | --- | --- | --- | --- | | cipher | 字符串 | 否 | 根据[TODO]定义的TLS加密套件描述 | | protocol | 字符串 | 否 | 取值为“TLS”、“TLSv1.2”或“TLSv1.3”之一 | | certificates | 对象数组 | 否 | 表示证书链的对象数组，第一个元素为根证书 | ### IP数据（ip_data数组中的元素） 以下为ip_data数组内每个元素包含的字段： | 字段名 | 字段类型 | 是否可为空 | 描述 | | --- | --- | --- | --- | | ip | 字符串 | 否 | IP地址 | | from_record | 字符串 | 否 | 捕获该地址的DNS记录类型 | | remarks | 对象 | 否 | Ping往返时间、“是否存活”标志以及RDAP/地理/ASN评估日期 | | rdap | 对象 | 是 | RDAP数据，格式与DNS RDAP字段类似，详见JSON Schema | | geo | 对象 | 是 | 来自GeoLite2 City数据库的地理定位数据（例如纬度、经度、城市、国家等） | | asn | 对象 | 是 | 来自GeoLite2 ASN数据库的自治系统数据（自治系统编号、组织、网络） | 致谢 感谢OpenPhish团队授权使用并发布其数据集，同时感谢VirusTotal为研究用途提供API访问权限。本研究得到捷克共和国内政部资助的Flow-based Encrypted Traffic Analysis项目（编号VJ02010024），以及布尔诺理工大学资助的Smart Information Technology for a Resilient Society项目（编号FIT-S-23-8209）的支持。