Geoparsing with Large Language Models: Leveraging the linguistic capabilities of generative AI to improve geographic information extraction

Geoparsing with Large Language Models The .zip file included in this repository contains all the code and data required to reproduce the results from our paper. Note, however, that in order to run the OpenAI models, users will required an OpenAI API key and sufficient API credits. Data The data used for the paper are in the datasetst and results folders. **Datasets: **This contains the XML files (LGL and Geovirus) and Json files (News2024) used to benchmark the models. It also contains all the data used to fine-tune the gpt-3.5 model, the prompt templates sent to the LLMs, and other data used for mapping and data creation. **Results: **This contains the results for the models on the three datastes. The folder is separated by dataset, with a single .csv file giving the results for each model on each dataset separately. The .csv file is structured so that each row contains either a predicted toponym and an associated true toponym (along with assigned spatial coordinates), if the model correctly identified a toponym; otherwise the true toponym columns are empty for false positives and the predicted columns are empty for false negatives. Code The code is split into two seperate folders gpt_geoparser and notebooks. **GPT_Geoparser: **this contains the classes and methods used process the XML and JSON articles (data.py), interact with the Nominatim API for geocoding (gazetteer.py), interact with the OpenAI API (gpt_handler.py), process the outputs from the GPT models (geoparser.py) and analyse the results (analysis.py). Notebooks: This series of notebooks can be used to reproduce the results given in the paper. The file names a reasonably descriptive of what they do within the context of the paper. Code/software Requirements Numpy Pandas Geopy Scitkit-learn lxml openai matplotlib Contextily Shapely Geopandas tqdm huggingface_hub Gnews Access information Other publicly accessible locations of the data: The LGL and GeoVirus datasets can also be obtained here (opens in new window). Abstract Geoparsing- the process of associating textual data with geographic locations - is a key challenge in natural language processing. The often ambiguous and complex nature of geospatial language make geoparsing a difficult task, requiring sophisticated language modelling techniques. Recent developments in Large Language Models (LLMs) have demonstrated their impressive capability in natural language modelling, suggesting suitability to a wide range of complex linguistic tasks. In this paper, we evaluate the performance of four LLMs - GPT-3.5, GPT-4o, Llama-3.1-8b and Gemma-2-9b - in geographic information extraction by testing them on three geoparsing benchmark datasets: GeoVirus, LGL, and a novel dataset, News2024, composed of geotagged news articles published outside the models' training window. We demonstrate that, through techniques such as fine-tuning and retrieval-augmented generation, LLMs significantly outperform existing geoparsing models. The best performing models achieve a toponym extraction F1 score of 0.985 and toponym resolution accuracy within 161 km of 0.921. Additionally, we show that the spatial information encoded within the embedding space of these models may explain their strong performance in geographic information extraction. Finally, we discuss the spatial biases inherent in the models' predictions and emphasize the need for caution when applying these techniques in certain contexts. Methods This contains the data and codes required to reproduce the results from our paper. The LGL and GeoVirus datasets are pre-existing datasets, with references given in the manuscript. The News2024 dataset was constructed specifically for the paper. To construct the News2024 dataset, we first created a list of 50 cities from around the world which have population greater than 1000000. We then used the GNews python package https://pypi.org/project/gnews/ (opens in new window) to find a news article for each location, published between 2024-05-01 and 2024-06-30 (inclusive). Of these articles, 47 were found to contain toponyms, with the three rejected articles referring to businesses which share a name with a city, and which did not otherwise mention any place names. We used a semi autonmous approach to geotagging the articles. The articles were first processed using a Distil-BERT model, fine tuned for named entity recognicion. This provided a first estimate of the toponyms within the text. A human reviewer then read the articles, and accepted or rejected the machine tags, and added any tags missing from the machine tagging process. We then used OpenStreetMap to obtain geographic coordinates for the location, and to identify the toponym type (e.g. city, town, village, river etc). We also flagged if the toponym was acting as a geo-political entity, as these were reomved from the analysis process. In total, 534 toponyms were identified in the 47 news articles.基于大语言模型（Large Language Model，LLM）的地理解析 本仓库包含的.zip文件涵盖了复现本文实验结果所需的全部代码与数据。不过需要注意的是，若要运行OpenAI模型，用户需具备OpenAI API密钥及足够的API额度。 ## 数据 本文实验所用数据存储于datasets与results文件夹中。 **数据集：** 本文件夹包含用于模型基准测试的XML文件（LGL与Geovirus）以及JSON文件（News2024），同时涵盖了用于微调gpt-3.5模型的全部数据、发送给大语言模型的提示模板，以及其他用于映射与数据生成的相关数据。 **实验结果：** 本文件夹包含三类数据集上的模型实验结果。该文件夹按数据集进行划分，每个数据集对应一个单独的.csv文件，用于展示各模型在对应数据集上的实验结果。.csv文件的结构为：若模型成功识别出地名（toponym），则每一行包含预测地名、对应的真实地名（以及分配的空间坐标）；若为假阳性，则真实地名列为空；若为假阴性，则预测地名列为空。 ## 代码 代码分为两个独立文件夹：gpt_geoparser与notebooks。 **GPT_Geoparser：** 本文件夹包含用于处理XML与JSON文本的类与方法（data.py）、与Nominatim地理编码API交互的代码（gazetteer.py）、与OpenAI API交互的代码（gpt_handler.py）、处理GPT模型输出的代码（geoparser.py）以及结果分析代码（analysis.py）。 **Notebooks：** 本系列笔记本可用于复现本文中的实验结果，文件名清晰说明了其在本文研究范围内的功能。 ## 代码/软件依赖 - Numpy - Pandas - Geopy - Scikit-learn - lxml - openai - matplotlib - Contextily - Shapely - Geopandas - tqdm - huggingface_hub - Gnews ## 数据获取途径 该数据的其他公开获取渠道如下： LGL与GeoVirus数据集可通过此链接获取（将在新窗口打开）。 ## 摘要 地理解析（Geoparsing）是指将文本数据与地理空间位置进行关联的过程，是自然语言处理领域的关键挑战之一。地理空间语言往往具有歧义性与复杂性，使得地理解析成为一项极具难度的任务，需要借助成熟的语言建模技术。近年来，大语言模型（Large Language Model，LLM）在自然语言建模领域展现出了卓越的性能，表明其适用于各类复杂的语言任务。本文中，我们通过在三个地理解析基准数据集上进行测试，评估了四款大语言模型——GPT-3.5、GPT-4o、Llama-3.1-8b以及Gemma-2-9b——在地理信息提取方面的性能。这三个数据集分别为GeoVirus、LGL，以及一款全新的数据集News2024，该数据集由模型训练窗口期之外发布的带地理标签的新闻文章组成。我们通过微调与检索增强生成（Retrieval-Augmented Generation）等技术，证明了大语言模型的性能显著优于现有的地理解析模型。性能最优的模型在地名提取任务上的F1值达到0.985，在距离真实位置161公里范围内的地名匹配准确率达到0.921。此外，我们发现这些模型嵌入空间中所编码的空间信息可以解释其在地理信息提取任务中的优异表现。最后，我们讨论了模型预测中固有的空间偏见，并强调了在特定场景下应用此类技术时需保持谨慎。 ## 实验方法 本部分涵盖了复现本文实验结果所需的数据与代码。LGL与GeoVirus数据集为已有公开数据集，相关引用已在正文中给出。News2024数据集是专为本文研究构建的全新数据集。 为构建News2024数据集，我们首先从全球范围内筛选出50个人口超过100万的城市列表。随后，我们使用GNews Python包（https://pypi.org/project/gnews/，将在新窗口打开）为每个城市检索一篇发布于2024年5月1日至2024年6月30日（含首尾日期）的新闻文章。其中，47篇文章被发现包含地名，剩余3篇文章仅提及与城市同名的企业，未提及其他任何地名，因此被排除。 我们采用半自动化的方式为文章添加地理标签：首先使用经过微调用于命名实体识别的Distil-BERT模型处理文章，初步识别文本中的地名；随后由人工审核员通读文章，确认或拒绝机器生成的标签，并补充机器识别遗漏的标签。之后，我们通过OpenStreetMap获取这些地点的地理坐标，并确定地名的类型（例如城市、城镇、村庄、河流等）。我们同时标记了作为地缘政治实体的地名，这类地名将被从分析流程中移除。最终，在47篇新闻文章中共识别出534个地名。