EU Public Consultations and Feedback Portal - Dataset of all public input data to EU legislation, attached files and extraction of textual responses, including automated translation to English (until May 2025)

EU Public Consultations and Feedback Portal - Dataset of all data, attached files and extraction of textual responses Purpose of repository This data repository contains all data (to the best of the author's knowledge) on public input to EU legislation publicly available through the EU Public Consultations and Feedback Portal https://ec.europa.eu/info/law/better-regulation as of May 2025. Through the EU Public Consultations and Feedback Portal, the broader public of the EU is asked to supply input to proposed regulation, via a feedback form field and/ or by attaching a file related to a specific consultation process instantiated by the EU. The data was downloaded from the portal via the Python package [eu_consultations](https://github.com/marioangst/eu_consultations) specifically developed for this purpose, which allows to download metadata on consultations, download attached files, extract texts from files and then store all information in a specific data format. All non-English texts were also translated into English using an automated process (source language is always retained). The aim of this dataset is to facilitate academic analysis of how the public participates in EU public consultations. > This is version v0.2.0 of the dataset. It includes the raw data, as well as extracted and translated texts for all initiatives. Repository structure The repository contains the raw data as scraped plus results of processing all documents using docling to extract textual contents from all documents that were possible to be processed (mainly .pdf and .docx files). It also contains the exact code run for scraping for documenting the procedure. Data Raw Data The downloaded data is in a zipped file named `v0_2_0_bytopic.zip`. The data is organized in folders by topic (the EU assigns every consultation to one of 38 overarching topics). In every <topic> folder, a subfolder labelled with the date (YYYY-MM-DD-Minute-Second) where the scraping process for the topic was initiated. It contains: `consultation_data.json`: metadata about all initiatives and consultations grouped below them in the topic`consultations_with_downloads.json`: The initiative data with download paths in the folder `files/` to all files that were downloaded for each consultationconsultations_with_extracted`: A massive .json file, which also contains the extracted text for every file in every attachment for every initiative. The benefit of these files is that they can be directly loaded with the eu_consultations Python package, using read_initiatives_from_json(), which reads the data in a validated format. However, all processed files are also contained in a `files/` folder. Further, for every processed file in `files/`, which follow the naming convention <fileid>_<filename>.<extension> the conversion to lossless docling JSON is stored in `files/docling/<fileid>.json`. Further `files/consultations` also include the metadata for every consultation within the topic separately, to facilitate working with single consultations. The folder `logs` contain the logs created during scraping of the data. Processed Data The repository also contains processed textual data in `v0_2_0_texts_by_initiative.zip` and metadata in `consultations_meta.csv`. - `consultations_meta.csv`: Metadata for all consultations to orient the processed output.- `extraction_rates.csv`: Detailed statistics on file extraction rates per topic.- `v0_2_0_texts_by_initiative.zip`: Contains the extracted and translated texts for every initiative, organized by initiative ID. - `<id>_feedback_form_texts.jsonl`: Extracted texts from feedback forms. - `<id>_feedback_form_texts_translated.jsonl`: Translated texts from feedback forms. - `<id>_feedback_attachment_texts.jsonl`: Extracted texts from attachments. - `<id>_feedback_attachment_texts_translated.jsonl`: Translated texts from attachments. Processed Data Structure The processed data is provided in JSONL format. **Feedback Form Texts (`<id>_feedback_form_texts.jsonl`)**```json{ "feedback_id": 837, "text": "The content of the feedback...", "consultation_id": 12345, "initiative_id": 1002}``` **Attachment Texts (`<id>_feedback_attachment_texts.jsonl`)**```json{ "attachment_id": "090166e5e3038a1f", "text": "Extracted text content...", "feedback_id": 837, "consultation_id": 12345, "initiative_id": 1002}``` Translated files (`*_translated.jsonl`) contain an additional `"text_en"` field with the English translation. Translation Process Translations were performed using the [Facebook SeamlessM4T v2 Large](https://huggingface.co/facebook/seamless-m4t-v2-large) model.The process involved detecting the language of the text and translating non-English text to English.This allows for consistent analysis across inputs in different languages. Metadata (`consultations_meta.csv`) This file contains metadata for consultations, linking them to initiatives.Key columns include:- `id`: Consultation ID (links to `consultation_id` in processed data)- `initiative_id`: Initiative ID (links to `initiative_id` in processed data)- `title`: Title of the consultation- `initiative_title`: Title of the initiative- `plannedStartDate`, `plannedEndDate`: Planned duration of the consultation Extraction Rates (`extraction_rates.csv`) This file provides transparency on the success rate of text extraction from attached files (PDF, DOC, DOCX).It lists the number of files downloaded, the number successfully processed by `docling`, and the resulting extraction rate per topic. Code The code that was used for the scraping procedure for the data is contained in a zipped file named `src.zip`. It contains crucially: - a `pyproject.toml` file to replicate the procedure using `uv`- a script `scrape_parallel.sh`, which was used to scrape topic by topic in parallel, first calling a script to check which topics were already finished and then running the scraping script `data_gathering/scrape_topic.py` to scrape a specific topic.- `data_processing/prepare_all_data.py`: Script to extract texts from the scraped raw data.- `data_processing/translate_folder.py`: Script to translate the extracted texts.