QCRI/LlamaLens-Hindi-Native

---
license: cc-by-nc-sa-4.0
task_categories:
- text-classification
language:
- hi
tags:
- Social Media
- News Media
- Sentiment
- Stance
- Emotion
pretty_name: "LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content -- Hindi-Native"
size_categories:
- 10K<n<100K
dataset_info:
- config_name: Sentiment_Analysis
splits:
- name: train
num_examples: 10039
- name: dev
num_examples: 1258
- name: test
num_examples: 1259
- config_name: MC_Hinglish1
splits:
- name: train
num_examples: 5177
- name: dev
num_examples: 2219
- name: test
num_examples: 1000
- config_name: Offensive_Speech_Detection
splits:
- name: train
num_examples: 2172
- name: dev
num_examples: 318
- name: test
num_examples: 636
- config_name: xlsum
splits:
- name: train
num_examples: 70754
- name: dev
num_examples: 8847
- name: test
num_examples: 8847
- config_name: Hindi-Hostility-Detection-CONSTRAINT-2021
splits:
- name: train
num_examples: 5718
- name: dev
num_examples: 811
- name: test
num_examples: 1651
- config_name: hate-speech-detection
splits:
- name: train
num_examples: 3327
- name: dev
num_examples: 476
- name: test
num_examples: 951
- config_name: fake-news
splits:
- name: train
num_examples: 8393
- name: dev
num_examples: 1417
- name: test
num_examples: 2743
- config_name: Natural_Language_Inference
splits:
- name: train
num_examples: 1251
- name: dev
num_examples: 537
- name: test
num_examples: 447
configs:
- config_name: Sentiment_Analysis
data_files:
- split: test
path: Sentiment_Analysis/test.json
- split: dev
path: Sentiment_Analysis/dev.json
- split: train
path: Sentiment_Analysis/train.json
- config_name: MC_Hinglish1
data_files:
- split: test
path: MC_Hinglish1/test.json
- split: dev
path: MC_Hinglish1/dev.json
- split: train
path: MC_Hinglish1/train.json
- config_name: Offensive_Speech_Detection
data_files:
- split: test
path: Offensive_Speech_Detection/test.json
- split: dev
path: Offensive_Speech_Detection/dev.json
- split: train
path: Offensive_Speech_Detection/train.json
- config_name: xlsum
data_files:
- split: test
path: xlsum/test.json
- split: dev
path: xlsum/dev.json
- split: train
path: xlsum/train.json
- config_name: Hindi-Hostility-Detection-CONSTRAINT-2021
data_files:
- split: test
path: Hindi-Hostility-Detection-CONSTRAINT-2021/test.json
- split: dev
path: Hindi-Hostility-Detection-CONSTRAINT-2021/dev.json
- split: train
path: Hindi-Hostility-Detection-CONSTRAINT-2021/train.json
- config_name: hate-speech-detection
data_files:
- split: test
path: hate-speech-detection/test.json
- split: dev
path: hate-speech-detection/dev.json
- split: train
path: hate-speech-detection/train.json
- config_name: fake-news
data_files:
- split: test
path: fake-news/test.json
- split: dev
path: fake-news/dev.json
- split: train
path: fake-news/train.json
- config_name: Natural_Language_Inference
data_files:
- split: test
path: Natural_Language_Inference/test.json
- split: dev
path: Natural_Language_Inference/dev.json
- split: train
path: Natural_Language_Inference/train.json
---

# LlamaLens: Specialized Multilingual LLM Dataset

## Overview

LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi.

<p align="center"> <img src="https://huggingface.co/datasets/QCRI/LlamaLens-Arabic/resolve/main/capablities_tasks_datasets.png" style="width: 40%;" id="title-icon"> </p>

## LlamaLens

This repo includes scripts needed to run our full pipeline, including data preprocessing and sampling, instruction dataset creation, model fine-tuning, inference and evaluation.

### Features

- Multilingual support (Arabic, English, Hindi)
- 18 NLP tasks with 52 datasets
- Optimized for news and social media content analysis

## 📂 Dataset Overview

### Hindi Datasets

| **Task** | **Dataset** | **# Labels** | **# Train** | **# Test** | **# Dev** |
| -------------------------- | ----------------------------------------- | ------------ | ----------- | ---------- | --------- |
| Cyberbullying | MC-Hinglish1.0 | 7 | 7,400 | 1,000 | 2,119 |
| Factuality | fake-news | 2 | 8,393 | 2,743 | 1,417 |
| Hate Speech | hate-speech-detection | 2 | 3,327 | 951 | 476 |
| Hate Speech | Hindi-Hostility-Detection-CONSTRAINT-2021 | 15 | 5,718 | 1,651 | 811 |
| Natural_Language_Inference | Natural_Language_Inference | 2 | 1,251 | 447 | 537 |
| Summarization | xlsum | -- | 70,754 | 8,847 | 8,847 |
| Offensive Speech | Offensive_Speech_Detection | 3 | 2,172 | 636 | 318 |
| Sentiment | Sentiment_Analysis | 3 | 10,039 | 1,259 | 1,258 |

---

## Results

Below, we present the performance of **L-Lens: LlamaLens** , where *"Eng"* refers to the English-instructed model and *"Native"* refers to the model trained with native language instructions. The results are compared against the SOTA (where available) and the Base: **Llama-Instruct 3.1 baseline**. The **Δ** (Delta) column indicates the difference between LlamaLens and the SOTA performance, calculated as (LlamaLens – SOTA).

---
| **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
|:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
| Factuality | fake-news | Mi-F1 | -- | 0.759 | 0.994 | 0.993 | -- |
| Hate Speech Detection | hate-speech-detection | Mi-F1 | 0.639 | 0.750 | 0.963 | 0.963 | 0.324 |
| Hate Speech Detection | Hindi-Hostility-Detection-CONSTRAINT-2021 | W-F1 | 0.841 | 0.469 | 0.753 | 0.753 | -0.088 |
| Natural Language Inference | Natural Language Inference | W-F1 | 0.646 | 0.633 | 0.568 | 0.679 | -0.078 |
| News Summarization | xlsum | R-2 | 0.136 | 0.078 | 0.171 | 0.170 | 0.035 |
| Offensive Language Detection | Offensive Speech Detection | Mi-F1 | 0.723 | 0.621 | 0.862 | 0.865 | 0.139 |
| Cyberbullying Detection | MC_Hinglish1 | Acc | 0.609 | 0.233 | 0.625 | 0.627 | 0.016 |
| Sentiment Classification | Sentiment Analysis | Acc | 0.697 | 0.552 | 0.647 | 0.654 | -0.050

## File Format

Each JSONL file in the dataset follows a structured format with the following fields:

- `id`: Unique identifier for each data entry.
- `original_id`: Identifier from the original dataset, if available.
- `input`: The original text that needs to be analyzed.
- `output`: The label assigned to the text after analysis.
- `dataset`: Name of the dataset the entry belongs.
- `task`: The specific task type.
- `lang`: The language of the input text.
- `instructions`: A brief set of instructions describing how the text should be labeled.

**Example entry in JSONL file:**

```
{
"id": "5486ee85-4a70-4b33-8711-fb2a0b6d81e1",
"original_id": null,
"input": "आप और बाकी सभी मुसलमान समाज के लिए आशीर्वाद हैं.",
"output": "not-hateful",
"dataset": "hate-speech-detection",
"task": "Factuality",
"lang": "hi",
"instructions": "Classify the given text as either 'not-hateful' or 'hateful'. Return only the label without any explanation, justification, or additional text."
}

```
## Model
[**LlamaLens on Hugging Face**](https://huggingface.co/QCRI/LlamaLens)

## Replication Scripts
[**LlamaLens GitHub Repository**](https://github.com/firojalam/LlamaLens)

## 📢 Citation

If you use this dataset, please cite our [paper](https://arxiv.org/pdf/2410.15308):

```
@article{kmainasi2024llamalensspecializedmultilingualllm,
title={LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content},
author={Mohamed Bayan Kmainasi and Ali Ezzat Shahroor and Maram Hasanain and Sahinur Rahman Laskar and Naeemul Hassan and Firoj Alam},
year={2024},
journal={arXiv preprint arXiv:2410.15308},
volume={},
number={},
pages={},
url={https://arxiv.org/abs/2410.15308},
eprint={2410.15308},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

LlamaLens is a multilingual language model designed specifically for analyzing and processing news and social media content. It supports 18 NLP tasks and uses 52 datasets across Arabic, English, and Hindi. The Hindi datasets include tasks such as cyberbullying detection, factuality, hate speech detection, natural language inference, text summarization, offensive language detection, and sentiment classification.

搜集汇总

背景与挑战

背景概述

该数据集是由Qatar Computing Research Institute（QCRI）开发的LlamaLens项目的一部分，专注于印地语（Hindi）的文本分类任务，涵盖仇恨言论检测、情感分析、事实核查等多个NLP任务。数据集规模在10万到100万条之间，基于社交媒体和新闻内容构建，旨在支持多语言LLM模型在新闻和社交媒体分析领域的训练和评估。数据集采用JSON格式，包含丰富的指令和标签信息，适用于研究者和开发者进行模型微调和性能测试。