NoseSense: Benchmarking Tool

# NoseSense ## Artifact Description The rapid evolution of Large Language Models (LLMs) has rendered traditional, static evaluation studies quickly obsolete upon publication. In this fast-paced landscape, evaluation methodologies must enable continuous and immediate assessment of newly released models. Although benchmarks exist for various software engineering tasks, there is a lack of targeted evaluation frameworks focusing on software test quality and the detection of test smells — design anti-patterns in test code that degrade maintainability and reliability. To address this limitation, we present **NoseSense**, a dynamic benchmarking tool designed to evaluate, in an automated and reproducible manner, the ability of LLMs to detect test smells in unit test code. NoseSense integrates a curated ground-truth oracle of 1,000 real-world Java code instances spanning 10 distinct test smell categories (validated using JNose). Built on a decoupled client-server architecture with a Python (FastAPI) backend and a React (Next.js) frontend, the platform enables the simultaneous evaluation of multiple AI models (both proprietary and open-source) through an asynchronous pipeline, offering real-time progress monitoring via Server-Sent Events (SSE). NoseSense features dynamic provider and model configuration via its interface, prompt engineering structured under the CRISPE framework with option randomization to neutralize positional bias, and an interactive analytical dashboard for macroscopic and per-category accuracy inspection. Results are automatically persisted in SQLite (`results.db`) and exported as CSV files (`results.csv`), providing a ready-to-use foundation for researchers and engineers. - **Demonstration video:** https://www.youtube.com/watch?v=D0o77YuM_Xs (also available on Zenodo: https://zenodo.org/records/20277258) --- ## Requested Badges - [x] **Available:** The artifact is publicly available in a GitHub repository and permanently archived on Zenodo with an assigned DOI (https://zenodo.org/records/20386683) under the open-source MIT License.- [x] **Functional:** The artifact is complete, well-documented, and includes clear, step-by-step instructions for installation and execution. It contains the full source code (backend and frontend), the prompt templates (`prompt.py`), the 1,000-sample test smell dataset (`test_smell_docs`), and pre-generated evaluation artifacts (`results.csv` and `results.db`) for full reproducibility and auditing. --- ## Required Skills To run and evaluate NoseSense, the following skills are required: - **Basic command-line interface (CLI) knowledge:** Executing terminal commands and running shell scripts.- **Basic Python and Node.js environment management:** Setting up virtual environments, installing dependencies via `pip`, and running client applications via `pnpm`.- **Basic API configuration knowledge:** Understanding how to obtain and configure API keys for LLM providers (e.g., OpenAI, Together AI, Groq) within the web interface. --- ## Required Resources To run the NoseSense artifact, the following resources are required: - **Operating System:** Compatible with Linux, macOS, and Windows.- **Software Dependencies:** Python 3.10+, Node.js 18+ with `pnpm` installed (`npm install -g pnpm`), and Git.- **Disk Space:** Minimum 500 MB of free disk space for the application, virtual environment, dependencies, and datasets.- **Network:** Active internet connection to send inference queries to remote LLM provider APIs. --- ## Repository & Artifact Package Contents The core components are structured as follows: ```textNoseSense/├── backend/ # Asynchronous Python Engine│ ├── core/│ │ └── prompt.py # CRISPE prompt templates and randomization logic│ ├── data/│ │ ├── test_smell_docs/ # Java test code samples (.txt files)│ │ ├── models/ # Provider configuration (models.json)│ │ └── output/ # Generated evaluation artifacts│ │ ├── results.csv # Exported dataset for analysis│ │ └── results.db # SQLite relational database│ ├── routes/ # API Gateway & SSE Endpoints│ ├── services/ # LLM Execution, Database & CSV services│ └── main.py # Application entry point└── frontend/ # Next.js Interactive Dashboard``` ### Artifact Files Breakdown **1. `prompt.py`** The prompt used by the tool to query the evaluated LLMs. **2. `test_smell_docs.zip`** Contains examples of test smells used as input for the benchmark — 1,000 real-world Java test code samples (100 per category across 10 distinct test smells), extracted and validated using the JNose static analysis tool. **3. `results.csv`** The benchmark results exported in CSV format, consolidating predictions, ground-truth labels, and per-instance metrics. **4. `results.db`** The benchmark results stored in a SQLite database, retaining relational records of executions, latency metrics, and misclassification patterns. --- ## Supported Test Smell Categories The benchmark suite evaluates models across 10 distinct test smell anti-patterns: | Category | Code | Description ||---|---|---|| **Assertion Roulette** | `AR` | A test method containing multiple assertions without explanatory messages. || **Conditional Test Logic** | `CTL` | Control flow structures (`if`, `for`, `while`) present in the test body. || **Duplicate Assert** | `DA` | Repeated assertions checking identical states or conditions within a test. || **Eager Test** | `ET` | A single test method attempting to verify multiple production methods. || **Exception Catching Throwing** | `ECT` | Explicit try-catch blocks used improperly instead of letting exceptions propagate. || **Ignored Test** | `IT` | Tests explicitly disabled or suppressed (e.g., via `@Ignore` annotation). || **Magic Number** | `MN` | Literal numbers used in assertions without clear semantic context. || **Sensitive Equality** | `SE` | Equality checks relying on object `toString()` string representations. || **Unknown Test** | `UT` | Test methods executing production code but lacking assertions altogether. || **Verbose Test** | `VT` | Excessively long or overly complex test method implementations. | --- ## Project Architecture **Backend (FastAPI + LangChain)** — `http://localhost:8001` Handles dynamic LLM instantiation, dataset loading, prompt execution, real-time SSE streaming, and result persistence to SQLite and CSV. **Frontend (Next.js + React)** — `http://localhost:3000` Interactive dashboard for managing AI providers and models, triggering benchmark runs, monitoring execution progress in real time via SSE, and exploring results through charts, confusion matrices, and detailed result tables. --- ## How to Install and Run ### Prerequisites - Python 3.10+- Node.js 18+ and `pnpm` (`npm install -g pnpm`)- API keys for your preferred LLM providers (configured via the UI). --- ### Quick Start (Recommended) Run the included startup script to build environments, install dependencies, spin up both servers, and open the web dashboard: ```bashchmod +x dev.sh./dev.sh``` --- ### Manual Setup #### 1. Running the Backend ```bashcd backend # Create and activate virtual environmentpython3 -m venv venvsource venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies and start serverpip install -r requirements.txtpython main.py``` *Backend runs on `http://localhost:8001`.* #### 2. Running the Frontend ```bashcd frontend # Install dependencies and start clientpnpm installpnpm run dev``` *Frontend runs on `http://localhost:3000`.* --- ## Configuring Providers and Models All model configurations are managed dynamically via the UI without requiring manual file edits: 1. **Add Provider**: Click **Add Provider**, input the provider name (e.g., "OpenAI", "Together"), select the API Type (`OpenAI-compatible`, `Google GenAI`, or `Anthropic`), and set your API key.2. **Set Custom Base URL**: For third-party OpenAI-compatible providers, supply the custom base URL (e.g., `https://api.together.xyz/v1`).3. **Add Models**: Add exact model identifiers (e.g., `gpt-4o`, `qwen/qwen2.5-7b-instruct-turbo`).4. **Execute**: Select desired models via checkboxes and click **Run Process**. Configurations are automatically persisted in `backend/data/models/models.json`: ```json{ "providers": { "openai": { "api_key": "sk-...", "api_type": "openai", "base_url": null, "models": ["gpt-4o", "gpt-4-turbo"] }, "together": { "api_key": "tg-...", "api_type": "openai", "base_url": "https://api.together.xyz/v1", "models": ["qwen/qwen2.5-7b-instruct-turbo"] } }}``` --- ## Technologies Used | Layer | Technologies ||---|---|| **Backend** | Python, FastAPI, LangChain, SQLite3 || **Frontend** | TypeScript, Next.js (App Router), React, Tailwind CSS || **Communication** | Server-Sent Events (SSE), REST || **Package Managers** | `pip` (Backend), `pnpm` (Frontend) | --- ## License This project and its accompanying artifact package are released under the **MIT License**.