Data from a article "Analise dos fatores econômicos, climáticos e científico sobre a produção de leite brasileira e suas diferenças regionais"

RBQL.csv CSV file presenting the names of cities and their respective latitude and longitude information for laboratories accredited by the Brazilian Milk Quality Network. embrapa.csv CSV file presenting the names of cities and their respective latitude and longitude information for Embrapa units scattered across Brazil. Dataset_final.csv The CSV file has 24 rows (one for each year from 2000 to 2023) and 25 columns (variables). It is a temporal dataset ranging from 2000 to 2023 column (year). Generally, it can be divided into 4 different categories, each possessing data for every region of Brazil: Regional milk production, column (average milk production); Climate data (temperature and precipitation) measured in degrees Celsius and millimeters respectively, columns (Average temperature and Average precipitation); Milk price data, involving average production price data, column (average milk price for the producer) and the average price of milk traded between producers, column (Average net milk price (between producers)). Finally, there is inflation variation data, column (ipca_variacao). quality_data.csv The CSV file contains a total of 4 Parameters CPP, CSS, EST, and ESD, grouped by state/region and by year from 2013 to 2023. The values presented are the maximum and minimum values of that respective year for each parameter. script_de_processamento_leite.ipynb The file presents the code developed using python notebooks, in which different libraries and processing steps are applied: the main tools used are: pandas and numpy: Data manipulation and aggregation. sklearn (KNNImputer): Used to fill in missing data (imputation), ensuring the final dataset has no gaps. prov: Library specialized in documenting data origin (provenance), generating flow diagrams. The script processes data in thematic blocks that correspond directly to the columns of your final CSV: A. Price Data (Regional) The script reads multiple CSV files from regional folders (South, Southeast, Center-West, North, Northeast). Renames original columns (such as "Minimum gross price") and calculates annual averages. Generates the columns: average milk price... and Average net price.... B. Climate Data (Temperature and Precipitation) Iterates over raw files from meteorological stations (likely from INMET). Extracts temperature and rain data, calculates annual averages/medians for each region. Generates the columns: Average temperature... and Average precipitation... for each region (SOUTH, SOUTHEAST, etc.). C. Production Data Loads a specific productivity file (producao). Generates the columns: average milk production... (millions). D. Economic Data Processes inflation files to calculate the annual variation. Generates the column: ipca_variacao. 3. Data Treatment Part of the script uses the KNN (K-Nearest Neighbors) algorithm to fill in null values. 4. Documentation and Provenance The end of the script (after CSV generation) is dedicated to creating "Data Provenance" diagrams. It visually maps where each piece of data came from (e.g., "INMET" -> "Climate Processing" -> "Final Dataset"), ensuring scientific traceability for the project. leite_visualization_script.ipynb Geospatial Analysis (Maps) The script uses the folium, geopandas, and HTML libraries to create interactive visualizations. Input Data: manually defines a dictionary coordenadas_cidades with latitudes and longitudes of Brazilian capitals (Rio Branco, Maceió, Macapá, etc.). Visualization: Generates maps pointing out locations, correlating Embrapa headquarters or milk quality centers with these coordinates. milk_production._map.html HTML file, which allows access to the constructed map and presents the dataset data dynamically, grouped by region. Furthermore, it presents information related to the geographic position of milk research and analysis centers (Embrapa and RBQL). Prov_xxx.png All files starting with prov_ present the provenance of each obtained datum in image format. The image presents information about the data acquisition and transformation process, from the location where it was obtained until being transformed into the dataframe format to be unified in the final dataset in the future. Specifically, quality_prov.png is the provenance graph of the quality dataset. requirements_script_leite.txt The file requirements_script_leite.txt presents information on the libraries used in the environment where the data processing and dataset construction script was executed. It is worth mentioning that the cited environment was Colab, so the main packages essential to run the file are presented in the produced article. requirements_map_script.txt The file requirements_script_leite.txt presents information on the libraries used in the environment where the script for the development of data visualization was executed.RBQL.csv：该CSV文件收录了巴西牛奶质量网络（Brazilian Milk Quality Network）认证实验室所在城市的名称，以及对应城市的经纬度信息。 embrapa.csv：该CSV文件收录了分布于巴西全境的巴西农业研究公司（Embrapa）下属机构所在城市的名称，以及对应城市的经纬度信息。 Dataset_final.csv：该CSV文件共包含24行数据（对应2000年至2023年的每一个年份）与25列变量，是一份以"year"（年份）列为时间维度、覆盖2000至2023年的时序数据集。该数据集可划分为4大类，每一类均涵盖巴西各区域的相关数据： - 区域牛奶产量数据：对应列"average milk production"（牛奶平均产量）； - 气候数据：包含以摄氏度为单位的气温数据与以毫米为单位的降水数据，对应列分别为"Average temperature"（平均气温）与"Average precipitation"（平均降水量）； - 牛奶价格数据：涵盖生产者牛奶平均生产价格（对应列"average milk price for the producer"），以及生产者之间交易的牛奶平均净价格（对应列"Average net milk price (between producers)"）； - 通胀变化数据：对应列"ipca_variacao"。 quality_data.csv：该CSV文件共包含CPP、CSS、EST、ESD共4项参数，数据按巴西各州/区域以及2013年至2023年的年份进行分组，展示了每一项参数在对应年份的最大值与最小值。 script_de_processamento_leite.ipynb：该文件为基于Python Notebook开发的代码脚本，集成了多种工具库与数据处理流程，核心使用的工具包括： - pandas与numpy：用于数据操作与聚合； - scikit-learn（sklearn）的KNNImputer：用于缺失值填充（插补），确保最终数据集无空值； - prov库：专门用于记录数据溯源（provenance），并生成流程流程图。 该脚本按主题模块处理数据，模块与最终CSV文件的列一一对应： A. 区域价格数据：脚本从南部、东南部、中西部、北部、东北部的区域文件夹中读取多个CSV文件，重命名原始列名（例如"Minimum gross price"）并计算年度平均值，最终生成"average milk price"与"Average net price"列。 B. 气候数据（气温与降水）：遍历气象站点的原始数据文件（大概率来自INMET（巴西国家气象研究所）），提取气温与降雨数据，计算各区域的年度平均值/中位数，最终为各区域（南部、东南部等）生成"Average temperature"与"Average precipitation"列。 C. 产量数据：加载特定的生产力文件"producao"，生成以百万为单位的"average milk production"列。 D. 经济数据：处理通胀相关文件以计算年度通胀变化，最终生成"ipca_variacao"列。 3. 数据处理：脚本中使用KNN（K近邻，K-Nearest Neighbors）算法填充空值。 4. 文档与数据溯源：脚本末尾（生成CSV文件后）专门用于创建"数据溯源"（Data Provenance）图表，以可视化方式映射每条数据的来源路径（例如"INMET → 气候数据处理 → 最终数据集"），保障项目的科学可追溯性。 leite_visualization_script.ipynb：地理空间分析（地图可视化）脚本，该脚本使用folium、geopandas与HTML库创建交互式可视化内容： - 输入数据：手动定义字典"coordenadas_cidades"，存储巴西各州首府的经纬度信息（例如里奥布朗库、马塞约、马卡帕等）； - 可视化效果：生成标注特定点位的地图，将巴西农业研究公司（Embrapa）下属机构或牛奶质量检测中心的位置与上述经纬度进行关联展示。 milk_production._map.html：该HTML文件可查看已构建的交互式地图，并按区域动态展示数据集内容，同时呈现牛奶研究与分析中心（Embrapa与RBQL）的地理分布信息。 Prov_xxx.png：所有以"prov_"开头的文件均为图像格式的数据溯源文件，展示了单条数据从采集源头到被转换为DataFrame格式、最终整合至最终数据集的完整获取与处理流程。其中，"quality_prov.png"为质量数据集的数据溯源图。 requirements_script_leite.txt：该文件列出了数据处理与数据集构建脚本运行环境中所需的全部库。需说明的是，本次运行环境为Colab，因此运行该脚本所需的核心包已在产出的文章中予以说明。 requirements_map_script.txt：该文件列出了数据可视化脚本开发过程中，运行环境所需的全部库。