Phishing Websites Dataset

The dataset consists of a collection of legitimate as well as phishing website instances. Each instance contains the URL and the relevant HTML page. The index.sql file is the root file, and it can be used to map the URLs with the relevant HTML pages. The dataset can serve as an input for the machine learning process. Highlights: - Total number of instances: 80,000 (83,275 instances in the dataset due to the existence of some removed SQL records in preprocessing stage) - Number of legitimate website instances (labelled as 0 in the SQL file): 50,000 - Number of phishing website instances (labelled as 1 in the SQL file): 30,000 Structure: The index.sql file is the root file. It consisted of five fields. 1). rec_id - record number 2). url - URL of the webpage 3). website - Filename of the webpage (i.e. 1635698138155948.html) 4). result - Indicates whether a given URL is phishing or not (0 for legitimate and 1 for phishing). 5). created_date - Webpage downloaded date Sources: - Legitimate Data [50,000] - These data were collected from two sources. 1). Google search - Simple keyword search on the google search engine was used, and the top 5 URLs of each search were collected. Domain restrictions were used and limited a maximum of 10 collections from a domain to have a diverse collection at the end. 2). Ebbu2017 Phishing Dataset [1] - Nearly 25,874 active URLs were collected from this repository - Phishing Data [30,000] - Three sources were used. 1). PhishTank - From 01 December 2020 to 31 October 2021 2). OpenPhish - From 29 September 2021 to 31 October 2021 3). PhishRepo [2] - From 29 September 2021 to 31 October 2021 Data Collection Process: - Legitimate Data: - The URLs were collected from the above sources and fetched the relevant webpages separately. - The URLs are in different lengths to minimize the URL lengths issue mentioned by Verma et al. [3]. - Phishing Data: - The URLs were collected from the above sources, and at the same time, the relevant web pages were fetched. - An automated script continuously monitored PhishTank and OpenPhish to collect the latest phishing URLs. - The collected URLs were fetched simultaneously to minimize the resource unavailable issue since the phishing pages do not exist for a longer period on the web. - PhishRepo provides all the resources relevant to a phishing webpage; therefore, simply use their download function to download PhishRepo data. References: [1]. Ebbu2017 Phishing Dataset. Accessed 31 October 2021. Available: https://github.com/ebubekirbbr/pdd/tree/master/input. [2]. PhishRepo. Accessed 31 October 2021. Available: https://moraphishdet.projects.uom.lk/phishrepo/. [3]. Verma, Rakesh M., Victor Zeng, and Houtan Faridi. "Data quality for security challenges: Case studies of phishing, malware and intrusion detection datasets.", 2019.

本数据集收录了合法网站与钓鱼网站（phishing website）两类实例。每条实例均包含对应的统一资源定位符（URL）与相关超文本标记语言（HTML）页面。index.sql为根文件，可用于建立URL与对应HTML页面的映射关系。本数据集可作为机器学习流程的输入数据。

### 数据集亮点
- 实例总规模：标注为80000条（因预处理阶段存在部分已移除的SQL记录，实际数据集中共包含83275条实例）
- 合法网站实例（SQL文件中标记为0）：50000条
- 钓鱼网站实例（SQL文件中标记为1）：30000条

### 数据集结构
index.sql为根文件，包含5个字段：
1. rec_id：记录编号
2. url：网页的URL
3. website：网页文件名（例如1635698138155948.html）
4. result：标识给定URL是否为钓鱼网站（0代表合法网站，1代表钓鱼网站）
5. created_date：网页下载日期

### 数据来源
- 合法数据（共50000条）：数据来自两个渠道
1. 谷歌搜索：通过谷歌搜索引擎执行简单关键词检索，采集每个检索结果的前5条URL，并设置域名访问限制，单个域名最多采集10条结果，以确保最终数据集的多样性。
2. Ebbu2017钓鱼数据集（Ebbu2017 Phishing Dataset）[1]：从该仓库中获取了近25874条活跃URL。
- 钓鱼数据（共30000条）：数据来自三个渠道
1. PhishTank：采集时间为2020年12月1日至2021年10月31日
2. OpenPhish：采集时间为2021年9月29日至2021年10月31日
3. PhishRepo[2]：采集时间为2021年9月29日至2021年10月31日

### 数据采集流程
- 合法数据：
从上述渠道采集URL后，分别抓取对应的网页。为最小化Verma等人[3]提出的URL长度偏差问题，本次采集的URL具备不同的长度。
- 钓鱼数据：
从上述渠道采集URL的同时，同步抓取对应的网页。通过自动化脚本持续监控PhishTank与OpenPhish平台，以获取最新钓鱼URL。由于钓鱼网页在互联网上的留存周期通常较短，因此对采集到的URL进行同步抓取，以降低资源不可用的问题发生率。PhishRepo提供了与钓鱼网页相关的全部资源，因此可直接使用其提供的下载功能获取该数据集的数据。

### 参考文献
[1] Ebbu2017钓鱼数据集（Ebbu2017 Phishing Dataset）。2021年10月31日访问。可获取：https://github.com/ebubekirbbr/pdd/tree/master/input.
[2] PhishRepo。2021年10月31日访问。可获取：https://moraphishdet.projects.uom.lk/phishrepo/.
[3] Verma, Rakesh M.、Victor Zeng与Houtan Faridi。《面向安全挑战的数据质量：以钓鱼、恶意软件与入侵检测数据集为例》，2019年。

搜集汇总

背景与挑战

背景概述

该数据集包含80,000个钓鱼和合法网站实例，用于机器学习研究，其中50,000个为合法网站，30,000个为钓鱼网站，数据来源多样，包括Google搜索和多个钓鱼网站数据库。