Dataset for paper Spotting the Hook: Leveraging Domain Data for Advanced Phishing Detection

The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS certificate fields, and GeoIP information for 432,572 benign domains from Cisco Umbrella and 68,353 phishing domains from PhishTank and OpenPhish services. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered as phishing by VT have been removed. The data was collected between March and November 2023.The final assessment of the data was conducted in December 2023.

The dataset is useful for statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing detection. 

Data Files

The data is located in two individual files:

benign.json - data for 432,572 benign domains, and

phishing.json - data for 68,353 phishing domains.

Data Structure

Both files contain a JSON array of records generated using mongoexport. The following table documents the structure of a record. Please note that:

some fields may be missing (they should be interpreted as nulls), 

extra fields may be present (they should be ignored), 

due to a processing error, the common_name field of the certificate objects always contains trailing symbols: ‘> .

Field name 

Field type 

Nullable 

Description 

domain_name 

String 

No 

The evaluated domain name 

url 

String 

No 

The source URL for the domain name 

evaluated_on 

Date 

No 

Date of last collection attempt 

source 

String 

No 

An identifier of the source 

sourced_on 

Date 

No 

Date of ingestion of the domain name 

dns 

Object 

Yes 

Data from DNS scan 

rdap 

Object 

Yes 

Data from RDAP or WHOIS 

tls 

Object 

Yes 

Data from TLS handshake 

ip_data 

Array of Objects 

Yes 

Array of data objects capturing the IP addresses related to the domain name 

DNS data (dns field) 

A 

Array of Strings 

No 

Array of IPv4 addresses 

AAAA 

Array of Strings 

No 

Array of IPv6 addresses 

TXT 

Array of Strings 

No 

Array of raw TXT values 

CNAME 

Object 

No 

The CNAME target and related IPs 

MX 

Array of Objects 

No 

Array of objects with the MX target hostname, priority and related IPs 

NS 

Array of Objects 

No 

Array of objects with the NS target hostname and related IPs 

SOA 

Object 

No 

All the SOA fields, present if found at the target domain name 

zone_SOA 

Object 

No 

The SOA fields of the target’s zone (closest point of delegation), present if found and not a record in the target domain directly 

dnssec 

Object 

No 

Flags describing the DNSSEC validation result for each record type 

ttls 

Object 

No 

The TTL values for each record type 

remarks 

Object 

No 

The zone domain name and DNSSEC flags 

RDAP data (rdap field) 

copyright_notice 

String 

No 

RDAP/WHOIS data usage copyright notice 

dnssec 

Bool 

No 

DNSSEC presence flag 

entitites 

Object 

No 

An object with various arrays representing the found related entity types (e.g. abuse, admin, registrant). The arrays contain objects describing the individual entities. 

expiration_date 

Date 

Yes 

The current date of expiration 

handle 

String 

No 

RDAP handle 

last_changed_date 

Date 

Yes 

The date when the domain was last changed 

name 

String 

No 

The target domain name for which the data in this object are stored 

nameservers 

Array of Strings 

No 

Nameserver hostnames provided by RDAP or WHOIS 

registration_date 

Date 

Yes 

First registration date 

status 

Array of Strings 

No 

The state of the registered object [TODO] 

terms_of_service_url 

String 

No 

URL of the RDAP usage ToS 

url 

String 

No 

URL of the RDAP entity 

whois_server 

String 

No 

WHOIS server address 

TLS data (tls field) 

cipher 

String 

No 

TLS cipher suite description according to [TODO] 

protocol 

String 

No 

One of “TLS”, ”TLSv1.2”, ”TLSv1.3” 

certificates 

Array of Objects 

No 

Array of objects representing the certificate chain, the first element is the root certificate 

IP data (elements in the ip_data array) 

ip  

String 

No 

The IP address 

from_record 

String 

No 

The type of the DNS record the address was captured from 

remarks 

Object 

No 

Ping round-trip time, “is alive” flag and rdap/geo/asn evaluation dates 

rdap 

Object 

Yes 

RDAP data, similar to DNS RDAP, see the JSON Schema for details 

geo 

Object 

Yes 

Geolocation data from the GeoLite2 City database (e.g. latitude, longitude, city, country, etc.) 

asn 

Object 

Yes 

Autonomous system data from the GeoLite2 ASN database (ASN, organization, network) 

Acknowledgements

We would like to thank the OpenPhish Team for grating permission to use and publish their dataset. We also thank VirusTotal for providing us access to the API for research purposes. The research has been supported by the Flow-based Encrypted Traffic Analysis project, no. VJ02010024, granted by the Ministry of the Interior of the Czech Republic and the Smart Information Technology for a Resilient Society project, no. FIT-S-23-8209, granted by Brno University of Technology.