PandasCorpus

Overview PandasCorpus is a dataset collected from the GitHub repositories that contains a large number of Pandas workflows and extracted code features. A workflow is the Pandas code that is written and shared in Jupyter notebooks. The dataset contains 139k Jupyter notebooks from roughly 100k unique repositories. Quickstart Open quickstart.ipynb for a step-by-step walkthrough of loading the data, inspecting metrics, resolving notebook hashes to GitHub URLs, and filtering cells by Pandas operation. For deeper analysis and visualizations, see stats.ipynb. Repository Contents The PandasCorpus contains the following dataset files (CSV or Parquet versions) Dataset: code_cells.csv (or .parquet.gzip): 3.8 million cells of code cells with fields: notebook_hash, cell_id, and source markdown_cells.csv (or .parquet.gzip): 1.4 million cells of markdown cells with the same fields as code_cells.csv Metrics and links: code_cells_metrics.csv: the metrics gathered from every code cell pandas_ops_count.csv: count of pandas operations over all cells code_links.csv: links to repositories and files of downloaded notebooks link_hash_map.csv: lookup table connecting code cells with code links API Operations and others: csv_count.csv: number of csv files in each repository csv_links.csv (or .parquet.gzip): links to csv files for each repository pandas_ops.csv: dataframe and general functions that output dataframe, series, or array-like types mpl_ops.csv: all plotting functions of the matplotlib library seaborn_ops.csv: all plotting functions of the seaborn library Workflow This is the pipeline to download and process all the data: search_notebooks.py: search GitHub for potential Jupyter notebooks download_notebooks.py: download notebooks from GitHub ipynb_to_csv.py: convert all notebooks into 2 dataframes db_to_csv.py: convert SQLite database to csv file analyze_code.py: analyze code cells, extracting metrics Additional programs: check_csv.py: check every repository for csv files parquet_converter.py: compress the dataframe from csv to parquet recreate_notebooks.py: recreates all notebooks from the dataset Tunable Parameters To obtain more or different results, these things can be tuned: pandas_ops.csv: currently it contains only 136 operations of Dataframe and general functions. It can be extended, for instance, with index objects, window objects, and so on. Query parameters (in search_notebooks.py): these parameters are already good, but can be extended.