zjunlp/ConceptEdit

---
license: cc-by-nc-sa-4.0
---

<div align="center">

**Editing Conceptual Knowledge for Large Language Models**

---

<p align="center">
<a href="#-conceptual-knowledge-editing">Overview</a> •
<a href="#-usage">How To Use</a> •
<a href="#-citation">Citation</a> •
<a href="https://arxiv.org/abs/2403.06259">Paper</a> •
<a href="https://zjunlp.github.io/project/ConceptEdit">Website</a>
</p>
</div>

## 💡 Conceptual Knowledge Editing

<div align=center>
<img src="./flow1.gif" width="70%" height="70%" />
</div>

### Task Definition

**Concept** is a generalization of the world in the process of cognition, which represents the shared features and essential characteristics of a class of entities.
Therefore, the endeavor of concept editing aims to modify the definition of concepts, thereby altering the behavior of LLMs when processing these concepts.

### Evaluation

To analyze conceptual knowledge modification, we adopt the metrics for factual editing (the target is the concept $C$ rather than factual instance $t$).

- `Reliability`: the success rate of editing with a given editing description
- `Generalization`: the success rate of editing **within** the editing scope
- `Locality`: whether the model's output changes after editing for unrelated inputs

Concept Specific Evaluation Metrics

- `Instance Change`: capturing the intricacies of these instance-level changes
- `Concept Consistency`: the semantic similarity of generated concept definition

## 🌟 Usage

### 🎍 Current Implementation
As the main Table of our paper, four editing methods are supported for conceptual knowledge editing.
| **Method** | GPT-2 | GPT-J | LlaMA2-13B-Chat | Mistral-7B-v0.1
| :--------------: | :--------------: | :--------------: | :--------------: | :--------------: |
| FT | ✅ | ✅ | ✅ | ✅ |
| ROME | ✅ | ✅ |✅ | ✅ |
| MEMIT | ✅ | ✅ | ✅| ✅ |
| PROMPT | ✅ | ✅ | ✅ | ✅ |

### 💻 Run
You can follow [EasyEdit](https://github.com/zjunlp/EasyEdit/edit/main/examples/ConceptEdit.md) to run the experiments.

## 📖 Citation

Please cite our paper if you use **ConceptEdit** in your work.

```bibtex
@misc{wang2024editing,
title={Editing Conceptual Knowledge for Large Language Models},
author={Xiaohan Wang and Shengyu Mao and Ningyu Zhang and Shumin Deng and Yunzhi Yao and Yue Shen and Lei Liang and Jinjie Gu and Huajun Chen},
year={2024},
eprint={2403.06259},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

## 🎉 Acknowledgement

We would like to express our sincere gratitude to [DBpedia](https://www.dbpedia.org/resources/ontology/)，[Wikidata](https://www.wikidata.org/wiki/Wikidata:Introduction)，[OntoProbe-PLMs](https://github.com/vickywu1022/OntoProbe-PLMs) and [ROME](https://github.com/kmeng01/rome).

Their contributions are invaluable to the advancement of our work.

原始信息汇总

💡 Conceptual Knowledge Editing

Task Definition

Concept 是对认知过程中世界的一种概括，代表一类实体的共享特征和本质特征。因此，概念编辑的目的是修改概念的定义，从而改变大型语言模型（LLMs）处理这些概念时的行为。

Evaluation

为了分析概念知识修改，我们采用了针对事实编辑的指标（目标是概念 $C$ 而不是事实实例 $t$）。

• Reliability: 使用给定编辑描述的编辑成功率
• Generalization: 在编辑范围内的编辑成功率
• Locality: 编辑后模型对无关输入的输出是否发生变化

概念特定评估指标

• Instance Change: 捕捉这些实例级变化的复杂性
• Concept Consistency: 生成的概念定义的语义相似性

🌟 Usage

🎍 Current Implementation

作为我们论文的主要表格，支持四种概念知识编辑方法。

Method	GPT-2	GPT-J	LlaMA2-13B-Chat	Mistral-7B-v0.1
FT	✅	✅	✅	✅
ROME	✅	✅	✅	✅
MEMIT	✅	✅	✅	✅
PROMPT	✅	✅	✅	✅

💻 Run

您可以按照 EasyEdit 运行实验。

📖 Citation

如果您在工作中使用 ConceptEdit，请引用我们的论文。

bibtex @misc{wang2024editing, title={Editing Conceptual Knowledge for Large Language Models}, author={Xiaohan Wang and Shengyu Mao and Ningyu Zhang and Shumin Deng and Yunzhi Yao and Yue Shen and Lei Liang and Jinjie Gu and Huajun Chen}, year={2024}, eprint={2403.06259}, archivePrefix={arXiv}, primaryClass={cs.CL} }