QCRI/LlamaLens-Hindi-Native

--- license: cc-by-nc-sa-4.0 task_categories: - text-classification language: - hi tags: - Social Media - News Media - Sentiment - Stance - Emotion pretty_name: "LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content -- Hindi-Native" size_categories: - 10K<n<100K dataset_info: - config_name: Sentiment_Analysis splits: - name: train num_examples: 10039 - name: dev num_examples: 1258 - name: test num_examples: 1259 - config_name: MC_Hinglish1 splits: - name: train num_examples: 5177 - name: dev num_examples: 2219 - name: test num_examples: 1000 - config_name: Offensive_Speech_Detection splits: - name: train num_examples: 2172 - name: dev num_examples: 318 - name: test num_examples: 636 - config_name: xlsum splits: - name: train num_examples: 70754 - name: dev num_examples: 8847 - name: test num_examples: 8847 - config_name: Hindi-Hostility-Detection-CONSTRAINT-2021 splits: - name: train num_examples: 5718 - name: dev num_examples: 811 - name: test num_examples: 1651 - config_name: hate-speech-detection splits: - name: train num_examples: 3327 - name: dev num_examples: 476 - name: test num_examples: 951 - config_name: fake-news splits: - name: train num_examples: 8393 - name: dev num_examples: 1417 - name: test num_examples: 2743 - config_name: Natural_Language_Inference splits: - name: train num_examples: 1251 - name: dev num_examples: 537 - name: test num_examples: 447 configs: - config_name: Sentiment_Analysis data_files: - split: test path: Sentiment_Analysis/test.json - split: dev path: Sentiment_Analysis/dev.json - split: train path: Sentiment_Analysis/train.json - config_name: MC_Hinglish1 data_files: - split: test path: MC_Hinglish1/test.json - split: dev path: MC_Hinglish1/dev.json - split: train path: MC_Hinglish1/train.json - config_name: Offensive_Speech_Detection data_files: - split: test path: Offensive_Speech_Detection/test.json - split: dev path: Offensive_Speech_Detection/dev.json - split: train path: Offensive_Speech_Detection/train.json - config_name: xlsum data_files: - split: test path: xlsum/test.json - split: dev path: xlsum/dev.json - split: train path: xlsum/train.json - config_name: Hindi-Hostility-Detection-CONSTRAINT-2021 data_files: - split: test path: Hindi-Hostility-Detection-CONSTRAINT-2021/test.json - split: dev path: Hindi-Hostility-Detection-CONSTRAINT-2021/dev.json - split: train path: Hindi-Hostility-Detection-CONSTRAINT-2021/train.json - config_name: hate-speech-detection data_files: - split: test path: hate-speech-detection/test.json - split: dev path: hate-speech-detection/dev.json - split: train path: hate-speech-detection/train.json - config_name: fake-news data_files: - split: test path: fake-news/test.json - split: dev path: fake-news/dev.json - split: train path: fake-news/train.json - config_name: Natural_Language_Inference data_files: - split: test path: Natural_Language_Inference/test.json - split: dev path: Natural_Language_Inference/dev.json - split: train path: Natural_Language_Inference/train.json --- # LlamaLens: Specialized Multilingual LLM Dataset ## Overview LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi. <p align="center"> <img src="https://huggingface.co/datasets/QCRI/LlamaLens-Arabic/resolve/main/capablities_tasks_datasets.png" style="width: 40%;" id="title-icon"> </p> ## LlamaLens This repo includes scripts needed to run our full pipeline, including data preprocessing and sampling, instruction dataset creation, model fine-tuning, inference and evaluation. ### Features - Multilingual support (Arabic, English, Hindi) - 18 NLP tasks with 52 datasets - Optimized for news and social media content analysis ## 📂 Dataset Overview ### Hindi Datasets | **Task** | **Dataset** | **# Labels** | **# Train** | **# Test** | **# Dev** | | -------------------------- | ----------------------------------------- | ------------ | ----------- | ---------- | --------- | | Cyberbullying | MC-Hinglish1.0 | 7 | 7,400 | 1,000 | 2,119 | | Factuality | fake-news | 2 | 8,393 | 2,743 | 1,417 | | Hate Speech | hate-speech-detection | 2 | 3,327 | 951 | 476 | | Hate Speech | Hindi-Hostility-Detection-CONSTRAINT-2021 | 15 | 5,718 | 1,651 | 811 | | Natural_Language_Inference | Natural_Language_Inference | 2 | 1,251 | 447 | 537 | | Summarization | xlsum | -- | 70,754 | 8,847 | 8,847 | | Offensive Speech | Offensive_Speech_Detection | 3 | 2,172 | 636 | 318 | | Sentiment | Sentiment_Analysis | 3 | 10,039 | 1,259 | 1,258 | --- ## Results Below, we present the performance of **L-Lens: LlamaLens** , where *"Eng"* refers to the English-instructed model and *"Native"* refers to the model trained with native language instructions. The results are compared against the SOTA (where available) and the Base: **Llama-Instruct 3.1 baseline**. The **Δ** (Delta) column indicates the difference between LlamaLens and the SOTA performance, calculated as (LlamaLens – SOTA). --- | **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** | |:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:| | Factuality | fake-news | Mi-F1 | -- | 0.759 | 0.994 | 0.993 | -- | | Hate Speech Detection | hate-speech-detection | Mi-F1 | 0.639 | 0.750 | 0.963 | 0.963 | 0.324 | | Hate Speech Detection | Hindi-Hostility-Detection-CONSTRAINT-2021 | W-F1 | 0.841 | 0.469 | 0.753 | 0.753 | -0.088 | | Natural Language Inference | Natural Language Inference | W-F1 | 0.646 | 0.633 | 0.568 | 0.679 | -0.078 | | News Summarization | xlsum | R-2 | 0.136 | 0.078 | 0.171 | 0.170 | 0.035 | | Offensive Language Detection | Offensive Speech Detection | Mi-F1 | 0.723 | 0.621 | 0.862 | 0.865 | 0.139 | | Cyberbullying Detection | MC_Hinglish1 | Acc | 0.609 | 0.233 | 0.625 | 0.627 | 0.016 | | Sentiment Classification | Sentiment Analysis | Acc | 0.697 | 0.552 | 0.647 | 0.654 | -0.050 ## File Format Each JSONL file in the dataset follows a structured format with the following fields: - `id`: Unique identifier for each data entry. - `original_id`: Identifier from the original dataset, if available. - `input`: The original text that needs to be analyzed. - `output`: The label assigned to the text after analysis. - `dataset`: Name of the dataset the entry belongs. - `task`: The specific task type. - `lang`: The language of the input text. - `instructions`: A brief set of instructions describing how the text should be labeled. **Example entry in JSONL file:** ``` { "id": "5486ee85-4a70-4b33-8711-fb2a0b6d81e1", "original_id": null, "input": "आप और बाकी सभी मुसलमान समाज के लिए आशीर्वाद हैं.", "output": "not-hateful", "dataset": "hate-speech-detection", "task": "Factuality", "lang": "hi", "instructions": "Classify the given text as either 'not-hateful' or 'hateful'. Return only the label without any explanation, justification, or additional text." } ``` ## Model [**LlamaLens on Hugging Face**](https://huggingface.co/QCRI/LlamaLens) ## Replication Scripts [**LlamaLens GitHub Repository**](https://github.com/firojalam/LlamaLens) ## 📢 Citation If you use this dataset, please cite our [paper](https://arxiv.org/pdf/2410.15308): ``` @article{kmainasi2024llamalensspecializedmultilingualllm, title={LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content}, author={Mohamed Bayan Kmainasi and Ali Ezzat Shahroor and Maram Hasanain and Sahinur Rahman Laskar and Naeemul Hassan and Firoj Alam}, year={2024}, journal={arXiv preprint arXiv:2410.15308}, volume={}, number={}, pages={}, url={https://arxiv.org/abs/2410.15308}, eprint={2410.15308}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```LlamaLens is a multilingual language model designed specifically for analyzing and processing news and social media content. It supports 18 NLP tasks and uses 52 datasets across Arabic, English, and Hindi. The Hindi datasets include tasks such as cyberbullying detection, factuality, hate speech detection, natural language inference, text summarization, offensive language detection, and sentiment classification.

搜集汇总

背景与挑战

背景概述

该数据集是由Qatar Computing Research Institute（QCRI）开发的LlamaLens项目的一部分，专注于印地语（Hindi）的文本分类任务，涵盖仇恨言论检测、情感分析、事实核查等多个NLP任务。数据集规模在10万到100万条之间，基于社交媒体和新闻内容构建，旨在支持多语言LLM模型在新闻和社交媒体分析领域的训练和评估。数据集采用JSON格式，包含丰富的指令和标签信息，适用于研究者和开发者进行模型微调和性能测试。