Type Hints in Python Libraries and Frameworks: An Empirical Analysis of Adoption and Maintenance

Python Type Annotation Research Dataset Type Annotation Usage and Inference Across Popular Python Repositories This repository contains research data, analysis scripts, and paper materials for a comprehensive study of Python type annotations in open-source software. Overview This research investigates: How frequently do developers introduce type annotations? (RQ1) What types of annotations are most commonly used? (RQ2-RQ3) Can LLMs infer types from unannotated code? (RQ4) Temporal trends in annotation adoption (RQ5) The study analyzed type annotations across 152 popular Python repositories, extracted ground-truth developer annotations, and compared them against blind Pyright type inference to evaluate type inference capabilities. Dataset Contents This release includes four main components: 1. Research Scripts (research_scripts.zip - 2.5 MB) Complete pipeline for: Repository discovery: src/top_repos.py Type extraction: src/type_extractor.py Type categorization: src/extract_type_annotations.py Metrics computation: src/type_extractor_metrics.py LLM inference pipeline: src/pipeline.py Research question analyses: src/rq*.py (RQ1-RQ10) Statistical analysis: src/h1_shared.py, hypothesis testing utilities Unit tests: tests/ All scripts are production-quality Python code with minimal dependencies (PyGithub, pandas, plotly). 2. Primary Research Data (research_data.zip - 4.8 GB) output_filtered/ (4.6 GB) Extracted type annotations from 152 Python repositories. One CSV per repository containing: file: Source file path member_name: Qualified name (e.g., MyClass.my_method.param_name) type: Type annotation string (or null if unannotated) member_type: parameter, return, or variable context_code: Surrounding source code context Repositories included: Data processing: DeepSpeed, PaddleNLP, PyMuPDF ML/AI: MetaGPT, NeMo, OpenRLHF Web frameworks: FastAPI, Django, Starlette Utilities: CustomTkinter, Pillow, TextBlob And 141 more (see repository list below) Each CSV is independently analyzable and can be loaded with: import pandas as pd df = pd.read_csv('output_filtered/DeepSpeed.csv') inference_type_comparison/ (415 MB) Pyright type inference results. One CSV per repository comparing: ground_truth_annotation: Developer-written type annotation predicted_annotation: Blindly inferred by Pyright (no source code hints) annotation_match: Whether predicted matches ground truth (normalized) mypy_ok: Whether predicted type passes mypy validation Used for RQ4 analysis: "Can LLMs predict developer type annotations?" 3. Analysis Outputs (analysis_outputs.zip - ~200 MB) Intermediate data used by research question analyses: annotation_history/ (3.0 MB) Historical annotation data per repository: Tracks when type annotations were introduced Used by RQ1: "Frequency of annotation changes" annotation_timeline/ (197 MB) Temporal evolution of type annotations: Timestamp data for annotation adoption Used by RQ5: "Temporal trends" 4. Paper Materials (paper_materials.zip - 30 MB) latex/figs/ - Publication figures: rq3_heatmap.pdf/png - Heatmap of type annotation distribution rq3_sankey.pdf/png - Sankey diagram of type category flows venn_inference.pdf/png - Venn diagram (developer vs Pyright inference) Data Flow and Relationships GitHub API ↓ Repository discovery (top_repos.py) ↓ Clone repositories → repos/ (local working copy) ↓ Extract type annotations (type_extractor.py) ↓ output/ (raw extracted annotations) ↓ Categorize types (extract_type_annotations.py) ↓ output_filtered/ ← PRIMARY DATA (4.6 GB) ↓ Type summary tables (type_summary_table.py) ↓ Analysis scripts (rq*.py) ↓ Published results & figures (latex/figs/) For inference (RQ4): output_filtered/ CSVs ↓ Strip annotations (strip_annotations.py) ↓ snippets/ (unannotated code index) ↓ Pyright inference (blind inference) ↓ inference_type_comparison/ ← RQ4 DATA (415 MB) Type Categorization Scheme All type annotations are categorized into four groups: Python Built-in Data Types Basic Python types: str, int, float, bool, list, dict, tuple, set, etc. Python Internal Objects From typing module: List[T], Dict[K,V], Optional[T], Union[A,B], Callable, etc. Local Objects Classes and types defined within the same repository External Objects Third-party library types (e.g., numpy.ndarray, pandas.DataFrame, library-specific exceptions) Repository List (152 repositories) Machine Learning & AI tensorflow, torch, jax, transformers, accelerate, openai, anthropic huggingface, metaGPT, nemo, deepspeed, axolotl, ollama Data Processing & Science pandas, numpy, scipy, scikit-learn, polars, dask paddleNLP, paddleSeg, paddleFormers, plotly Web Frameworks & APIs django, flask, fastapi, starlette, aiohttp, websockets requests, httpx, graphql-core NLP & Text Processing nltk, spacy, textblob, gensim, allennlp, nlpaug Computer Vision pillow, opencv, imageio, scikit-image, albumentations, fastSAM Development Tools pytest, sphinx, mypy, pydantic, sqlalchemy, celery poetry, jupyter, ipython, black, pylint Utilities & Other requests, boto3, click, typer, loguru, tqdm serializers, api clients, benchmarks (See IC_repos.csv for complete list with GitHub links) Usage Instructions Setup # Install dependencies pip install -r requirements.txt # Configure GitHub API token (optional, for repo discovery) cp .env.example .env # Edit .env and add your GITHUB_TOKEN Analyzing Type Annotations Load and analyze raw type annotations: import pandas as pd # Load Django's type annotations django_types = pd.read_csv('output_filtered/Django.csv') # Count annotated vs unannotated members print(django_types['type'].value_counts()) # Filter to parameters only params = django_types[django_types['member_type'] == 'parameter'] print(f"Total parameters: {len(params)}") print(f"Annotated parameters: {(params['type'] != 'null').sum()}") Running Research Question Analyses Compute statistics for individual RQs: # RQ1: Frequency of annotation changes python src/rq1_plots.py # RQ2: Type usage patterns python src/rq2.py # RQ3: Type category distribution python src/rq3_heatmap.py # RQ4: LLM inference comparison python src/rq4_analysis.py # Generate RQ4 Venn diagram (developer vs Pyright) python src/generate_rq4_venn.py Reproducibility All scripts accept the data as-is: No additional preprocessing required All paths in scripts are relative (run from repo root) Dependencies: pandas, PyGithub, plotly, matplotlib, seaborn Extending the Analysis To add new analyses: Load CSVs from output_filtered/ Reference type categorization in src/extract_type_annotations.py Use src/h1_shared.py utilities for common operations File Locations Directory Contents Size Purpose output_filtered/ Type annotations (152 repos) 4.6 GB Primary data source inference_type_comparison/ Pyright inference results 415 MB RQ4 analysis annotation_history/ Historical annotation data 3.0 MB Temporal analysis annotation_timeline/ Timeline data 197 MB RQ1-5 plots hypothesis/ Violin plots & statistics 2.2 MB H1 hypothesis tests src/ Python scripts ~200 KB Complete pipeline latex/ Paper & figures 4.2 MB Publication materials tests/ Unit tests ~100 KB Validation Key Metrics 152 repositories analyzed 3,874,520 members (parameters, returns, variables) extracted 1,247,816 annotated (32.2% coverage) 414 external type libraries identified Pyright inference accuracy (RQ4): See inference_type_comparison/ results Related Work This research builds on: Type4Py: Type inference for Python (Pradel et al.) Pyright: Static type checker for Python (Microsoft) Pyre: Type inference for Python (Facebook/Meta) Mypy: Static type checker for Python Contact & Questions For questions or issues: Check paper for methodology details See src/ docstrings for individual script usage Generated: 2026-07-25 Repository: Python Type Annotation Research Paper: [Submitted to IST]