EDGAR Financial Data Pipeline

EDGAR Financial Data Pipeline — ACR Research (Dutta & Talukdar, 2026) Collects patent-linked financial panel data for Fortune 500 firms in Information Technology, Pharmaceuticals, and Financial Services, 2009–2025, supporting the manuscript "Resetting the Clock: Adaptive Complexity Reconfiguration as a Mechanism for Perpetual Competitive Advantage." This deposit contains the data-collection scripts, intermediate files, final analysis panel, and the regression/figure outputs reported in the manuscript. Note on scope: This pipeline covers Stages 1, 2, and 5 (CIK mapping, XBRL financial pull, and panel assembly) for the 2009–2025 window used in the manuscript. Stage 3 and Stage 4 scripts (early 10-K download and parsing, originally written for a 1995–2015 extension) are included for reference but were not run for this deposit — their outputs (early_financials_panel.csv, download_log_10k.csv, the raw_10k/ directory) are not part of this release. Files in This Deposit Scripts File Purpose 01_cik_mapping.py Stage 1: firm name → SEC CIK mapping 02_xbrl_financial_pull.py Stage 2: XBRL financial data pull, 2009–2025 03_early_10k_download.py Stage 3 (reference only, not run for this release): early 10-K download, 1995–2008 04_early_10k_parse.py Stage 4 (reference only, not run for this release): early 10-K parsing, 1995–2008 05_panel_assembly.py Stage 5: merge and dependent-variable construction Input / Intermediate Data File Description acr_candidate_sample_2009_2025.csv Candidate firm list (input to Stage 1) cik_mapping.csv Firm → CIK mapping table (Stage 1 output) cik_mapping_REVIEW.csv Low-confidence CIK matches flagged for manual review xbrl_financials_panel_2009_2025.csv Raw XBRL financials pull, 2009–2025 (Stage 2 output) year_missing.csv Firm-years with missing financial data Final Analysis Files File Description acr_financial_panel_FINAL.csv Analysis-ready financial panel (Stage 5 output) sustained_advantage_spells.csv Sustained-advantage spell table (dependent variable construction) industry_median_roa_by_year.csv Industry-year median ROA benchmarks used to construct industry-adjusted ROA Manuscript Results Files File Corresponds to table1_descriptives.csv Table 1, descriptive statistics table2_regressions.csv Table 2, OLS regression results table4_iv_results.csv Table 3 (instrumental variable estimates) figure2_kaplan_meier.png Figure 4, Kaplan–Meier survival curves figure3_moderation.png Figure 2, moderation by competitive intelligence capability figure4_inverted_u.png Figure 3, reconfiguration scope vs. advantage period Note: the file table4_iv_results.csv corresponds to Table 3 in the manuscript (instrumental variable estimates), and the figure files are numbered independently of the manuscript's figure numbers. Please double-check this mapping against the final typeset manuscript before publishing this README, and relabel the files or this table if the numbering doesn't match. Setup Install Python 3.10+ if not already installed. Create a virtual environment (recommended): python -m venv venv source venv/bin/activate # Mac/Linux venv\Scripts\activate # Windows Install dependencies: pip install requests pandas rapidfuzz tqdm beautifulsoup4 lxml sec-edgar-downloader Place all .py files and acr_candidate_sample_2009_2025.csv in the same folder. How to Reproduce the Final Panel Run the following stages in order. Each stage saves its output as a CSV, so you can stop and restart at any stage. Stage 1 — CIK mapping python 01_cik_mapping.py Input: acr_candidate_sample_2009_2025.csv Creates: cik_mapping.csv, cik_mapping_REVIEW.csv Pause here: open cik_mapping_REVIEW.csv and manually correct any firms with confidence < 80. Look up their CIK at SEC EDGAR company search. Add corrections to the MANUAL_OVERRIDES dict in 01_cik_mapping.py and re-run until cik_mapping_REVIEW.csv is empty or acceptable. Stage 2 — XBRL financial pull python 02_xbrl_financial_pull.py Creates: xbrl_financials_panel_2009_2025.csv SEC rate limit is respected (see below); runtime scales with the number of firms in cik_mapping.csv. Safe to re-run if interrupted; already-fetched firms are skipped if a checkpoint is added. Firm-years with missing data are logged to year_missing.csv. Stage 5 — Panel assembly python 05_panel_assembly.py Inputs: xbrl_financials_panel_2009_2025.csv Creates: acr_financial_panel_FINAL.csv, sustained_advantage_spells.csv, industry_median_roa_by_year.csv Stages 3 and 4 (03_early_10k_download.py, 04_early_10k_parse.py) are not part of the 2009–2025 reproduction path above. They are included in this deposit for reference in case the panel is later extended backward to 1995–2008, but running them is not required to reproduce acr_financial_panel_FINAL.csv as released here. Data Quality Checks (run after Stage 5) import pandas as pd df = pd.read_csv("acr_financial_panel_FINAL.csv") # 1. Check observation count print(f"Rows: {len(df)}") print(f"Firms: {df['cik'].nunique()}") # 2. Check for implausible ROA values print(df['roa'].describe()) # Should be roughly -0.5 to +0.5 # 3. Check coverage by year print(df.groupby('fiscal_year')['roa'].count()) # 4. Check industry balance print(df.groupby('industry_bucket')['cik'].nunique()) # 5. Identify firms with gaps in their panel (missing years) expected = df.groupby('cik').apply( lambda g: set(range(int(g['fiscal_year'].min()), int(g['fiscal_year'].max()) + 1)) ) actual = df.groupby('cik')['fiscal_year'].apply(set) gaps = {cik: expected[cik] - actual[cik] for cik in expected.index if expected[cik] - actual[cik]} print(f"Firms with year gaps: {len(gaps)}") Note: I was not able to independently verify the row and firm counts in the released acr_financial_panel_FINAL.csv against the manuscript's reported sample (104 firms, 1,586 firm-year observations). Please run the checks above and confirm the printed counts match the manuscript before treating this deposit as the final archival record. Known Issues & Workarounds 1. Private/mutual firms (e.g., Liberty Mutual, Nationwide, State Farm) These firms don't file with the SEC. For them, use: Annual reports from their websites (PDF, then extract manually) Insurance regulatory filings (NAIC IRIS data — free) Supplement with SimFin where available 2. Financial firms (SIC 60xx–63xx) Insurance companies and banks use different financial statement formats. "Total assets" is still Assets, but "revenue" may appear as "Net premiums earned" (insurance) or "Net interest income" (banks). For firms with low coverage, augment using FDIC call reports (banks, free at https://www.ffiec.gov/npw/) or NAIC data (insurers). 3. Missing firm-years Firm-years that could not be retrieved or matched are logged in year_missing.csv. Check this file before treating panel gaps as data errors — some are expected (e.g., firms not yet public, or delisted mid-window). SEC Rate Limit Compliance The SEC requires: A User-Agent header with name + email (set in each script) Max 10 requests per second Respectful crawling (this pipeline uses a 0.15s pause ≈ 6 req/sec) Do not run multiple instances of these scripts simultaneously. Next Steps After This Pipeline Once acr_financial_panel_FINAL.csv is ready, the following steps (not included in this deposit) construct the remaining manuscript variables: Run the patent network pipeline (PatentsView + NBER data) to construct the annual modularity Q time series → ACR Frequency variable. Run structural break detection (Bai–Perron) on the Q series. Construct the Rival Decoding Signal Index (inventor mobility + patent topology convergence + product launch alignment). Merge all components into the final analysis dataset used for Tables 1–4 and Figures 2–4. License Creative Commons Attribution 4.0 International (CC-BY 4.0). Contact Questions: [e-mail]