Public Visibility of Student Engagement Analytics Governance in Universities: Replication Package

Replication package for a QS 2026 top-1000 public-document audit of student engagement analytics and learning analytics governance in universities. The package includes the institutional denominator frame, final validated corpus metadata, downloaded public source documents, extracted text, sanitized validation decisions, blind human validation results for mechanism coding, frozen structured annotation outputs, prompt and JSON-schema files, analysis inputs, and scripts for deterministic replay and live end-to-end LLM rerun. Two workflows are provided. 1. Deterministic replay from frozen model outputs. This is the exact-results workflow for the published tables and figures. 2. End-to-end live LLM rerun. This re-extracts text from the downloaded source documents, chunks the text, sends the stored prompts/schema to the OpenAI API, parses the responses, and rebuilds the LLM-dependent tables. Live LLM results may differ from the frozen published outputs because hosted models and sampling infrastructure can change. ## Contents - `data/qs1000_frame.csv`: denominator frame. - `data/final_corpus_metadata.csv`: final validated corpus metadata with relative file links. - `data/source_documents/`: downloaded public source documents used in the final corpus. - `data/extracted_text/`: extracted text for each included document. - `data/validation/`: sanitized human validation decisions and the blind validation benchmark for mechanism coding. - `data/llm_annotations/`: frozen structured annotation outputs used for deterministic mechanism analysis. - `data/analysis_inputs/`, `data/robustness/`, `data/extended/`, `data/topics/`: analysis inputs required to reproduce manuscript tables and checks. - `prompts/`: LLM coding prompt template and JSON response schema. - `scripts/reproduce_deterministic.py`: deterministic replay from frozen outputs. - `scripts/reproduce_end_to_end.py`: live LLM rerun from downloaded source documents. ## One-Line Installation ```bash python3 -m venv .venv && . .venv/bin/activate && python -m pip install -r requirements.txt ``` ## One-Line Run Commands ```bash bash scripts/reproduce.sh deterministic ``` ```bash OPENAI_API_KEY=YOUR_KEY bash scripts/reproduce.sh end-to-end --model gpt-5.4 ``` For a low-cost smoke test of the live LLM workflow: ```bash OPENAI_API_KEY=YOUR_KEY bash scripts/reproduce.sh end-to-end --model gpt-5.4 --limit 5 ``` Outputs are written to `results/deterministic/` or `results/end_to_end/`.