EU Public Consultations and Feedback Portal - Dataset of all public input data to EU legislation, attached files and extraction of textual responses (until May 2025)

Purpose of repository This dataset contains all data (to the best of the author's knowledge) on public input to EU legislation publicly available through the EU Public Consultations and Feedback Portal https://ec.europa.eu/info/law/better-regulation as of May 2025. Through the EU Public Consultations and Feedback Portal, the broader public of the EU is asked to supply input to proposed regulation, via a feedback form field and/ or by attaching a file related to a specific consultation process instantiated by the EU. The data was downloaded from the portal via the Python package eu_consultations specifically developed for this purpose, which allows to download metadata on consultations, download attached files, extract texts from files and then store all information in a specific data format. The aim of this dataset is to facilitate academic analysis of how the public participates in EU public consultations. 🚧 This is version v0.1.0 of the dataset. In a next version, the dataset will include a more accessible format for the textual data, as well as the results of a currently running translation process for all textual data. 🚧 Repository structure The repository contains the raw data as scraped plus results of processing all documents using docling to extract textual contents from all documents that were possible to be processed (mainly .pdf and .docx files). It also contains the exact code run for scraping for documenting the procedure. Data The downloaded raw data from scraping is in a zipped file named `v0_1_0_bytopic.zip`. The data is organized in folders by topic (the EU assigns every consultation to one of 38 overarching topics). In every <topic> folder, a subfolder labelled with the date (YYYY-MM-DD-Minute-Second) where the scraping process for the topic was initiated. It contains: `consultation_data.json`: metadata about all initiatives and consultations grouped below them in the topic `consultations_with_downloads.json`: The initiative data with download paths in the folder `files/` to all files that were downloaded for each consultation consultations_with_extracted`: A massive .json file, which also contains the extracted text for every file in every attachment for every initiative. The benefit of these files is that they can be directly loaded with the eu_consultations Python package, using read_initiatives_from_json(), which reads the data in a validated format. However, all processed files are also contained in a `files/` folder. Further, for every processed file in `files/`, which follow the naming convention <fileid>_<filename>.<extension> the conversion to lossless docling JSON is stored in `files/docling/<fileid>.json`. Further `files/consultations` also include the metadata for every consultation within the topic separately, to facilitate working with single consultations. The folder `logs` contain the logs created during scraping of the data. Code The code that was used for the scraping procedure for the data is contained in a zipped file named `src.zip`. It contains crucially: a `pyproject.toml` file to replicate the procedure using `uv` a script scrape_parallel.sh, which was used to scrape topic by topic in parallel, first calling a script to check which topics were already finished and then running the scraping script `data_gathering/scrape_topic.py` to scrape a specific topic.## 数据集仓库用途 本数据集收录了截至2025年5月，通过欧盟公共咨询与反馈门户（EU Public Consultations and Feedback Portal，https://ec.europa.eu/info/law/better-regulation）公开可获取的、所有与欧盟立法公共意见相关的数据（据作者所知已涵盖全部公开数据）。 欧盟通过该门户向欧盟广大公众征集立法提案意见，民众可通过反馈表单字段提交内容，或上传与欧盟发起的特定咨询流程相关的文件。 本数据集通过专为该场景开发的Python包`eu_consultations`从门户下载获取数据，该工具支持下载咨询元数据、上传附件，并从文件中提取文本，最终将所有信息以特定数据格式存储。 本数据集的设计目标是为学界开展公众参与欧盟公共咨询的相关学术分析提供支撑。 🚧 本数据集当前版本为v0.1.0。下一版本将提供更便于使用的文本数据格式，同时包含当前正在进行的全量文本数据翻译工作的成果。🚧 ## 仓库结构 本仓库包含爬取得到的原始数据，以及使用docling对可处理文档（主要为.pdf和.docx格式文件）提取文本内容后的处理结果。 同时收录了用于爬取的完整代码，以复现并记录整个流程。 ## 数据 爬取得到的原始数据存储于名为`v0_1_0_bytopic.zip`的压缩文件中。 数据集按主题分类存储（欧盟将每项咨询划分为38个一级主题之一）。 在每个<主题>文件夹下，存在一个以爬取该主题的启动时间（格式为YYYY-MM-DD-时分秒）命名的子文件夹，其中包含： `consultation_data.json`：该主题下所有立法倡议及所属咨询项目的元数据。 `consultations_with_downloads.json`：包含各咨询项目下载附件的存储路径（位于`files/`文件夹下）的立法倡议数据。 `consultations_with_extracted`：大型JSON文件，存储了所有立法倡议的所有附件文件的提取文本内容。 上述文件的优势在于可直接通过`eu_consultations` Python包的`read_initiatives_from_json()`函数加载，该函数会以经过校验的格式读取数据。 不过所有处理后的文件也存储于`files/`文件夹中。对于`files/`内遵循`<文件ID>_<文件名>.<扩展名>`命名规范的处理文件，其无损docling格式的转换结果将存储于`files/docling/<fileid>.json`中。 此外，`files/consultations`文件夹还单独存储了该主题下每项咨询的元数据，以方便对单个咨询项目进行操作。 `logs`文件夹存储了数据爬取过程中生成的日志文件。 ## 代码 用于数据爬取的代码存储于名为`src.zip`的压缩文件中，其中核心内容包括： 一份`pyproject.toml`文件，可通过`uv`工具复现整个爬取流程。 脚本`scrape_parallel.sh`用于并行按主题爬取数据：该脚本会先调用其他脚本检查已完成的主题，再运行爬取脚本`data_gathering/scrape_topic.py`以爬取指定主题。