A Dataset of Information (DNS, IP, WHOIS/RDAP, TLS, GeoIP) for a Large Corpus of Benign, Phishing, and Malware Domain Names 2024

The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS handshakes and certificates, and GeoIP information for 368,956 benign domains from Cisco Umbrella, 461,338 benign domains from the actual CESNET network traffic, 164,425 phishing domains from PhishTank and OpenPhish services, and 100,809 malware domains from various sources like ThreatFox, The Firebog, MISP threat intelligence platform, and other sources. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered malicious by VT have been removed from phishing and malware datasets. Similarly, benign domain names that were considered risky by VT have been removed from the benign datasets. The data was collected between March 2023 and July 2024. The final assessment of the data was conducted in August 2024.

The dataset is useful for cybersecurity research, e.g. statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing and malware website detection.

The dataset was created using software available in the associated GitHub repository nesfit/domainradar-dib.

Data Files

The data is located in the following individual files:

benign_umbrella.json - data for 368,956 benign domains from Cisco Umbrella,

benign_cesnet.json - data for 461,338 benign domains from the CESNET network,

phishing.json - data for 164,425 phishing domains, and

malware.json - data for 100,809 malware domains.

The schema.json file contains a JSON Schema with detailed description of the data entries.

Data Structure

Both files contain a JSON array of records generated using mongoexport (in the MongoDB Extended JSON (v2) format in Relaxed Mode). The following table documents the structure of a record. Please note that:

some fields may be missing (they should be interpreted as nulls), 

extra fields may be present (they should be ignored).

Field name 

Field type 

Nullable 

Description 

domain_name 

String 

No 

The evaluated domain name 

url 

String 

No 

The source URL for the domain name 

evaluated_on 

Date 

No 

Date of last collection attempt 

source 

String 

No 

An identifier of the source 

sourced_on 

Date 

No 

Date of ingestion of the domain name 

dns 

Object 

Yes 

Data from DNS scan 

rdap 

Object 

Yes 

Data from RDAP or WHOIS 

tls 

Object 

Yes 

Data from TLS handshake 

ip_data 

Array of Objects 

Yes 

Array of data objects capturing the IP addresses related to the domain name 

malware_type

String

No

The malware type/family or “unknown” (only present in malware.json) 

DNS data (dns field) 

A 

Array of Strings 

No 

Array of IPv4 addresses 

AAAA 

Array of Strings 

No 

Array of IPv6 addresses 

TXT 

Array of Strings 

No 

Array of raw TXT values 

CNAME 

Object 

No 

The CNAME target and related IPs 

MX 

Array of Objects 

No 

Array of objects with the MX target hostname, priority and related IPs 

NS 

Array of Objects 

No 

Array of objects with the NS target hostname and related IPs 

SOA 

Object 

No 

All the SOA fields, present if found at the target domain name 

zone_SOA 

Object 

No 

The SOA fields of the target’s zone (closest point of delegation), present if found and not a record in the target domain directly 

dnssec 

Object 

No 

Flags describing the DNSSEC validation result for each record type 

ttls 

Object 

No 

The TTL values for each record type 

remarks 

Object 

No 

The zone domain name and DNSSEC flags 

RDAP data (rdap field) 

copyright_notice 

String 

No 

RDAP/WHOIS data usage copyright notice 

dnssec 

Bool 

No 

DNSSEC presence flag 

entitites 

Object 

No 

An object with various arrays representing the found related entity types (e.g. abuse, admin, registrant). The arrays contain objects describing the individual entities. 

expiration_date 

Date 

Yes 

The current date of expiration 

handle 

String 

No 

RDAP handle 

last_changed_date 

Date 

Yes 

The date when the domain was last changed 

name 

String 

No 

The target domain name for which the data in this object are stored 

nameservers 

Array of Strings 

No 

Nameserver hostnames provided by RDAP or WHOIS 

registration_date 

Date 

Yes 

First registration date 

status 

Array of Strings 

No 

The state of the registered object (see RFC 7483, section 10.2.2)

terms_of_service_url 

String 

No 

URL of the RDAP usage ToS 

url 

String 

No 

URL of the RDAP entity 

whois_server 

String 

No 

WHOIS server address 

TLS data (tls field) 

cipher 

String 

No 

TLS cipher suite description according to IANA

protocol 

String 

No 

One of “TLS”, ”TLSv1.2”, ”TLSv1.3” 

certificates 

Array of Objects 

No 

Array of objects representing the certificate chain, the first element is the root certificate 

IP data (elements in the ip_data array) 

ip  

String 

No 

The IP address 

from_record 

String 

No 

The type of the DNS record the address was captured from 

remarks 

Object 

No 

Ping round-trip time, “is alive” flag and rdap/geo/asn evaluation dates 

rdap 

Object 

Yes 

RDAP data, similar to DNS RDAP, see the JSON Schema for details 

geo 

Object 

Yes 

Geolocation data from the GeoLite2 City database (e.g. latitude, longitude, city, country, etc.) 

asn 

Object 

Yes 

Autonomous system data from the GeoLite2 ASN database (ASN, organization, network) 

Acknowledgements

We would like to thank the OpenPhish Team for grating permission to use and publish their dataset. We also thank VirusTotal for providing us access to the API for research purposes.

This dataset includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

The research has been supported by the Flow-based Encrypted Traffic Analysis project, no. VJ02010024, granted by the Ministry of the Interior of the Czech Republic and the Smart Information Technology for a Resilient Society project, no. FIT-S-23-8209, granted by Brno University of Technology.