Innovating with Large Language Models at the UK Data Service for Census Information Discovery and Retrieval using Natural Language: Training, Evaluation and API Reference Data, 2024-2025

This collection contains research datasets and supporting API metadata developed and used by the ESRC-funded CORDIAL-AI project to train and evaluate generative-AI and agentic systems for the natural-language discovery and retrieval of UK census flow data. The research datasets comprise: a human-annotated dataset of realistic natural-language data requests paired with validated structured requests for the UK Data Service census flow data API; a synthetically generated dataset created through methods including paraphrasing, substitution of geographical areas and variables, and recombination of query structures to increase linguistic, geographical and topical coverage; and a separate test dataset used to evaluate the interpretation of natural-language requests and the construction of valid structured API queries. The records include natural-language requests, corresponding API representations, relevant identifiers and labels, and annotation and provenance information. The datasets were used to fine-tune smaller open-weight language models and assess the feasibility of domain adaptation, the computational and financial costs of local deployment, and the quality and accuracy of the resulting API requests. The test set supported evaluation of tasks including series and family selection, theme and topic identification, dataset selection, variable matching and geographical alignment. The collection also includes JSON snapshots of supporting metadata extracted from the census flow data API. These cover the 2011 open and safeguarded series and the 2021 open series; the three main census flow dataset families; all themes and topics available through the API at the time of extraction; relevant dataset metadata; and geography metadata associated with the 2011 and 2021 datasets. These API-derived files preserve the metadata context in which the human-annotated, synthetic and test records were created and evaluated. They support interpretation of the identifiers and labels used in the query-API pairs and improve reproducibility if the structure or contents of the live API change over time. Complete lists of individual geographical areas are not included because of their complexity and size. The collection documentation identifies the relevant API endpoint, parameters and geographical coverage required to retrieve these records. The collection contains metadata describing safeguarded census flow datasets but does not contain safeguarded census observations, census microdata or personal data. The API representations identify requested datasets and parameters but do not reproduce underlying census values. CORDIAL-AI was funded through the ESRC Future Data Services pilot programme to investigate how generative artificial intelligence could make complex research data easier to discover and retrieve while maintaining the accuracy, provenance, transparency and accountability expected of public data services. Led by the UCL Department of Information Studies in collaboration with UCL Advanced Research Computing and closely aligned with the UK Data Service, the project ran from September 2024 to December 2025. The project focused on UK census flow data, which describe movements between origins and destinations, including migration and travel to work or study. These data have considerable value for research and policy analysis but are particularly difficult to navigate because they are distributed across multiple datasets, geographical structures, variables and access levels. Their complexity can create substantial barriers for researchers and analysts, particularly those without specialist knowledge of census data structures. CORDIAL-AI developed and evaluated a working prototype that enables users to express data requirements in natural language. The system interprets a user’s request, identifies relevant datasets and metadata, and constructs a structured query for the UKDS census flow data API. A multi-agent architecture divides this process into specialised stages, including intent recognition, metadata search, dataset filtering, geography and variable selection, and API-query construction. Where information is missing or ambiguous, the system can make its assumptions explicit or request clarification from the user. A central design principle was the separation of language interpretation from authoritative data operations. Generative AI is used to interpret the user’s intent and assist with constructing valid requests, while all data retrieval, filtering and aggregation are undertaken deterministically by the underlying API. The language models do not receive the underlying census data: they operate on metadata and structured prompts, while the requested results are returned directly from the authoritative data service. Users can inspect the selected datasets, generated API queries, source metadata and intermediate filtering stages, supporting verification, provenance and reproducibility. The project produced a wider set of technical and methodological outputs to support the systematic development of AI-assisted data services. These included human-curated and synthetically augmented query-API datasets; pipelines for model training, fine-tuning and evaluation; task-specific benchmarking methods; synthetic versions of safeguarded datasets for safe experimentation; and enhancements to the census flow data API, metadata structures, and search and indexing infrastructure. The team experimented with proprietary cloud-based models and locally hosted open-weight models, examining trade-offs between performance, cost, governance and deployability.