qwq-misguided-attention

# Overview

This dataset presents responses from the [QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview) model to the [MisguidedAttention](https://github.com/cpldcpu/MisguidedAttention) prompt challenge, a collection of carefully crafted prompts designed to test LLMs' reasoning abilities in the presence of misleading information.

## About Misguided Attention

The Misguided Attention challenge consists of modified versions of well-known thought experiments, riddles, and paradoxes. These modifications are subtle yet significant, requiring careful step-by-step logical analysis rather than pattern matching from training data.

The challenge explores an interesting phenomenon: despite their computational nature, LLMs often exhibit a behavior similar to the human cognitive bias known as the Einstellungseffekt - where familiar patterns trigger learned responses, even when they're not appropriate for the modified problem at hand.

When presented with these prompts, an ideal response would demonstrate:

- Careful analysis of the specific problem details
- Step-by-step logical reasoning
- Recognition of how the problem differs from its classic version
- Arrival at the correct solution for the modified scenario

However, we often observe that models:

- Fall back on memorized solutions to the original problems
- Mix conflicting reasoning patterns
- Fail to notice crucial differences in the modified versions

## Creating Your Own Dataset

You can easily create similar datasets using the [observers](https://github.com/cfahlgren1/observers) package. Here's a minimal example:

```python
from observers.observers import wrap_openai
from observers.stores import DatasetsStore
from openai import OpenAI

# Initialize the OpenAI client
client = OpenAI(api_key="your-api-key")

# Configure dataset storage
store = DatasetsStore(
repo_name="your-dataset",
every=1 # sync frequency in minutes
)

# Wrap the client to automatically collect responses
wrapped_client = wrap_openai(client, store)

# Use the wrapped client as normal - responses will be saved automatically
response = wrapped_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Your prompt here"}]
)
```

## Using Hugging Face Serverless API

This dataset answers were generated using Hugging Face's Serverless Inference API, which provides OpenAI-compatible endpoints for various open-source models. This means you can use the standard OpenAI client library to interact with Hugging Face models:

```python
from openai import OpenAI

client = OpenAI(
base_url="https://api-inference.huggingface.co/v1/",
api_key="your-hf-token" # Get your token from Hugging Face
)

response = client.chat.completions.create(
model="Qwen/QwQ-32B-Preview", # Any HF model supporting chat
messages=[
{
"role": "system",
"content": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step.",
},
{"role": "user", "content": "Hello!"},
]
)
```

You can explore available models and try them interactively in the [Hugging Face Playground](https://huggingface.co/playground).

# 概览

本数据集收录了[QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview)模型针对[MisguidedAttention](https://github.com/cpldcpu/MisguidedAttention)（误导注意力挑战赛）提示词挑战赛的响应结果；该挑战赛由一系列精心设计的提示词组成，旨在测试大语言模型（Large Language Model，LLM）在存在误导性信息场景下的推理能力。

## 关于误导注意力挑战赛

该挑战赛包含对知名思想实验、谜题与悖论的修改版本，这些修改虽细微却至关重要，需要模型进行细致的逐步逻辑分析，而非依赖训练数据中的模式匹配。

该挑战赛旨在探究一项有趣的现象：尽管大语言模型具备计算属性，但其往往会展现出与人类「定势效应（Einstellungseffekt）」相似的认知偏差——即熟悉的模式会触发习得性响应，即便该响应并不适用于当前修改后的问题。

当面对这类提示词时，理想的响应应具备以下特征：
- 对问题的具体细节展开细致分析
- 开展逐步递进的逻辑推理
- 识别该问题与其经典版本的差异所在
- 针对修改后的场景得出正确解决方案

但实际观测中，我们常发现模型存在以下问题：
- 退而使用针对原始问题的记忆化解决方案
- 混杂相互矛盾的推理模式
- 未能留意修改版本中的关键差异

## 自定义数据集构建

你可借助`observers`工具包轻松构建同类数据集。以下为极简示例：

python
from observers.observers import wrap_openai
from observers.stores import DatasetsStore
from openai import OpenAI

# 初始化OpenAI客户端
client = OpenAI(api_key="your-api-key")

# 配置数据集存储
store = DatasetsStore(
repo_name="your-dataset",
every=1 # 同步频率，单位为分钟
)

# 包装客户端以自动收集响应结果
wrapped_client = wrap_openai(client, store)

# 按常规方式使用包装后的客户端——响应将自动保存
response = wrapped_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Your prompt here"}]
)

## 使用Hugging Face无服务器推理API

本数据集的响应结果通过Hugging Face的无服务器推理API生成，该API为各类开源模型提供与OpenAI兼容的接口端点。这意味着你可通过标准的OpenAI客户端库与Hugging Face模型进行交互：

python
from openai import OpenAI

client = OpenAI(
base_url="https://api-inference.huggingface.co/v1/",
api_key="your-hf-token" # 从Hugging Face获取你的令牌
)

response = client.chat.completions.create(
model="Qwen/QwQ-32B-Preview", # 任意支持聊天功能的Hugging Face模型
messages=[
{
"role": "system",
"content": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step.",
},
{"role": "user", "content": "Hello!"},
]
)

你可在[Hugging Face Playground](https://huggingface.co/playground)中浏览可用模型并进行交互式试用。