dleemiller/wiki-sim

Wiki Sim数据集是一个半合成数据集，源自`wikimedia/wikipedia`。每行包含1-3个从原始数据集中提取的参考句子，并使用优化的DSPy程序生成4种不同类型的相似句子：同义词替换、改写、概念重叠和上下文意义。数据集旨在扩展小型模型（如WordLlama）和通用嵌入模型的训练，并针对stsb和相似性任务进行优化。数据集包含多个子集，如`pair-score`、`pair-score-hard`、`triplet`、`triplet-hard`和`raw`。

This is a semi-synthetic dataset derived from `wikimedia/wikipedia`. Each row contains 1-3 reference sentences extracted from the original dataset. For each reference sentence, an optimized DSPy program generates 4 similar sentences, including synonym replacement, paraphrasing, conceptual overlap, and contextual meaning. Additionally, each result is scored using `cross-encoder/stsb-roberta-large`, and hard negatives are mined from different contiguous sentences in the original passage. The dataset aims to expand training for small models like WordLlama and general embedding models, targeting benchmarks like stsb and similarity tasks. The dataset columns include synonym, paraphrase, conceptual_overlap, contextual_meaning, reference, negative, negative_score, model_id, cross_encoder, synonym_score, paraphrase_score, conceptual_overlap_score, and contextual_meaning_score. The dataset also includes multiple subsets such as `pair-score`, `pair-score-hard`, `triplet`, `triplet-hard`, and `raw`.