siavava/ai-tech-articles

---
language:
- en
license: mit
task_categories:
- text-generation
- feature-extraction
pretty_name: AI/Technology Articles
tags:
- temporal series data
- language data
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
dataset_info:
features:
- name: id
dtype: int64
- name: year
dtype: int64
- name: title
dtype: string
- name: url
dtype: string
- name: text
dtype: string
splits:
- name: train
num_bytes: 180820047
num_examples: 17092
download_size: 81702923
dataset_size: 180820047
---

# AI/Tech Dataset

This dataset is a collection of AI/tech articles scraped from the web.

It's hosted on [HuggingFace Datasets](https://huggingface.co/datasets/siavava/ai-tech-articles), so it is easier to load in and work with.

## To load the dataset

### 1. Install [HuggingFace Datasets](https://huggingface.co/docs/datasets/installation.html)

```bash
pip install datasets
```

### 2. Load the dataset

```python
from datasets import load_dataset

dataset = load_dataset("siavava/ai-tech-articles")

# optionally, convert it to a pandas dataframe:
df = dataset["train"].to_pandas()
```

You do not need to clone this repo.
HuggingFace will download the dataset for you, the first time that you load it,
and cache it locally so it does not need to re-download it again
(unless it detects a change upstream).

## File Structure

- [`analytics.ipynb`](analytics.ipynb) - Notebook containing some details about the dataset.
- [`example.ipynb`](example.ipynb) - A minimal notebook that loads in the dataset and converts to Pandas.
- [`raw.csv`](raw.csv) - The raw data, in CSV format.
- `data/*.parquet`- compressed [parquet](https://www.databricks.com/glossary/what-is-parquet) containing the data.
- For raw text files, see the [scraper repo](https://github.com/siavava/scrape.hs) on GitHub.

原始信息汇总

数据集概述

基本信息

• 名称: AI/Technology Articles
• 语言: 英语
• 许可证: MIT
• 任务类别:
  • 文本生成
  • 特征提取
• 标签:
  • 时间序列数据
  • 语言数据

配置

• 默认配置:
  • 数据文件路径:
    • 训练集: data/train-*

数据集信息

• 特征:
  • id: 整数型
  • year: 整数型
  • title: 字符串型
  • url: 字符串型
  • text: 字符串型
• 分割:
  • 训练集:
    • 字节数: 180820047
    • 示例数: 17092
• 下载大小: 81702923
• 数据集大小: 180820047