How Type Hints are Used in Python Libraries and Frameworks: Early Insights

Dataset Overview projects_discovery.zipIncludes all auxiliary files used during dataset construction and during the generation of the CSV files contained in repositories_dataset.zip. clone_repos.py – Clones all repositories listed in top_repos.txt into the source directory. The official repository list used in the study is provided in top_repos_official.txt. create_repo_links.py / extract_repo_links.py – Utility scripts responsible for normalizing, cleaning, and generating correct GitHub repository paths for Python projects. filter_repos_annotations.py – Filters repositories based on a minimum threshold of type-annotation coverage. Only projects meeting or exceeding the specified annotation percentage are retained. get_about.py – Extracts the “About” and descriptive metadata from GitHub repositories to support repository categorization. merge_topics.py – Merges topic metadata from multiple CSV sources into a unified structure. type_hints.zipContains all scripts responsible for extracting, aggregating, and processing type-annotation information. type_extractor.py – Recursively scans all Python files inside each repository under repos/. For every file, it records all detected type annotations and outputs a CSV located at output/{repository}.csv. Each row contains:["file", "member_name", "type", "member_type", "context_code"]. type_extractor_metrics.py – Generates a per-repository summary from the extractor output. Produces a summary.csv including:["repo", "total", "typed_total", "total_typed_%", "variables_total", "variables_typed", "variables_typed_%", "parameters_total", "parameters_typed", "parameters_typed_%", "returns_total", "returns_typed", "returns_typed_%"].A final GLOBAL row aggregates all repositories to provide overall statistics. strip_annotations.py – Produces multi-level code snippets (local, medium, full scope) for all annotated regions. Each repository receives a CSV containing the extracted snippets. These files are used for evaluating small LLMs on automatic type-inference tasks. delimiter.py / drop_duplicates.py – Utility scripts for data cleaning, duplicate removal, and consistent CSV formatting. pipeline.py - Contains a small pipeline where each snippet is inputted into a small size LLM model, deepseek-r1:7b in this case, for automatic type inference performance. type_summary.py - Type to extract and create a category for each type seem at each repository unique csv file. add_category_column.py - Helper file to append the category column at the summary dataset. resulting_dataset_sheets/Contains the primary consolidated CSV files used for analysis in the paper. repos_merged.csv – Integrates GitHub repository URLs, classification (library vs. non-library), and descriptive metadata such as repository topics and “About” fields. summary.csv – Merges all computed metrics required to answer each research question. repositories_dataset/A reduced sample of the full 136-GB dataset. Each file corresponds to the output of type_extractor.py for one repository.Columns are: file – File path of the Python source where the annotation was found. member_name – Identifier of the function, variable, or member containing the annotation. type – The explicitly declared type annotation. context_code – The code snippet surrounding the annotation. The complete dataset (not fully uploaded due to size constraints) contains the full set of extracted annotations across all analyzed repositories.数据集概览 projects_discovery.zip 包含数据集构建过程中，以及生成repositories_dataset.zip内CSV文件所需的全部辅助文件。 clone_repos.py – 将top_repos.txt中列出的所有仓库克隆至源目录。本研究使用的官方仓库列表已在top_repos_official.txt中提供。 create_repo_links.py / extract_repo_links.py – 用于规范化、清理并为Python项目生成合规GitHub仓库路径的实用脚本。 filter_repos_annotations.py – 基于类型注解覆盖率的最小阈值过滤仓库，仅保留达到或超过指定注解占比的项目。 get_about.py – 从GitHub仓库中提取“About”信息及描述性元数据，用于辅助仓库分类。 merge_topics.py – 将多CSV来源的主题元数据合并为统一结构。 type_hints.zip 包含所有用于提取、聚合并处理类型注解信息的脚本。 type_extractor.py – 递归扫描repos/目录下每个仓库内的所有Python文件，记录所有检测到的类型注解，并将结果输出至output/{repository}.csv文件中。每行数据包含："["file", "member_name", "type", "member_type", "context_code"]"。 type_extractor_metrics.py – 从类型提取器的输出结果生成单仓库汇总统计信息，生成summary.csv文件，其中包含："["repo", "total", "typed_total", "total_typed_%", "variables_total", "variables_typed", "variables_typed_%", "parameters_total", "parameters_typed", "parameters_typed_%", "returns_total", "returns_typed", "returns_typed_%"]"。此外，最终会生成一条全局（GLOBAL）行汇总所有仓库的整体统计数据。 strip_annotations.py – 为所有带注解的代码区域生成多层级代码片段（局部、中等范围、全范围），每个仓库对应一个包含提取片段的CSV文件。这些文件用于评估小型大语言模型（Large Language Model, LLM）的自动类型推断任务性能。 delimiter.py / drop_duplicates.py – 用于数据清理、重复项移除及CSV格式统一的实用脚本。 pipeline.py – 包含一个小型推理流水线，将每个代码片段输入至本次实验使用的小型大语言模型deepseek-r1:7b中，以开展自动类型推断性能测试。 type_summary.py – 对每个仓库的专属CSV文件中出现的所有类型进行提取，并为其创建分类标签。 add_category_column.py – 用于向汇总数据集追加分类列的辅助脚本。 resulting_dataset_sheets/ 包含本论文分析所用的主要整合型CSV文件。 repos_merged.csv – 整合了GitHub仓库URL、分类信息（库项目与非库项目）以及描述性元数据（如仓库主题和“About”字段内容）。 summary.csv – 合并了所有用于回答各项研究问题所需的计算指标。 repositories_dataset/ 完整数据集的精简采样版本（完整数据集大小达136 GB，因体积限制未完全上传），每个文件对应type_extractor.py为单个仓库生成的输出结果。各列信息如下： file – 发现注解的Python源文件路径 member_name – 包含该注解的函数、变量或成员的标识符 type – 显式声明的类型注解 context_code – 注解所在的上下文代码片段 完整数据集包含所有被分析仓库的全部提取注解集合（因体积过大未完全上传）。