Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use

This repository contains transcriptions and evaluation metrics from Automatic Speech Recognition (ASR) experiments of the work "Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use" using OpenAI's Whisper models on audio files from the Gigaspeech dataset. If you use or base your work on this project or you need more details, please cite the following article: @ARTICLE{11186205, author={Arriaga, Carlos and Pozo, Alejandro and Conde, Javier and Alonso, Alvaro}, journal={IEEE Internet Computing}, title={Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use}, year={2025}, volume={29}, number={5}, pages={17-24}, keywords={Delays;Real-time systems;Measurement;Hardware;Mathematical models;Transformers;Time measurement;Speech to text;Loss measurement;Artificial intelligence}, doi={10.1109/MIC.2025.3614363}} Overview The dataset provides a comprehensive comparison of Whisper ASR performance across different: Model sizes: Tiny, Base, and Large Transcription modes: Batch (offline) and Real-time (online) Audio splitting algorithms: VAD, Fixed-length (2s, 3s), and Feedback-based The dataset includes 1,748 transcription files from 159 unique audio files processed under 11 different experimental configurations, along with detailed evaluation metrics. Dataset Structure . ├── transcriptions/ # Transcription outputs │ ├── batchBase/ # Batch mode with Whisper Base model │ ├── batchLarge/ # Batch mode with Whisper Large model │ ├── batchTiny/ # Batch mode with Whisper Tiny model │ ├── realBase2s/ # Real-time Base model with 2-second fixed chunks │ ├── realBase3s/ # Real-time Base model with 3-second fixed chunks │ ├── realBaseFeed/ # Real-time Base model with feedback-based splitting │ ├── realBaseVad/ # Real-time Base model with VAD-based splitting │ ├── realTiny2s/ # Real-time Tiny model with 2-second fixed chunks │ ├── realTiny3s/ # Real-time Tiny model with 3-second fixed chunks │ ├── realTinyFeed/ # Real-time Tiny model with feedback-based splitting │ └── realTinyVad/ # Real-time Tiny model with VAD-based splitting └── results/ # Evaluation metrics ├── batchBase.csv # Metrics for batch Base model ├── batchLarge.csv # Metrics for batch Large model ├── batchTiny.csv # Metrics for batch Tiny model ├── realBase2s.csv # Metrics for real-time Base with 2s chunks ├── realBase3s.csv # Metrics for real-time Base with 3s chunks ├── realBaseFeed.csv # Metrics for real-time Base with feedback ├── realBaseVad.csv # Metrics for real-time Base with VAD ├── realTiny2s.csv # Metrics for real-time Tiny with 2s chunks ├── realTiny3s.csv # Metrics for real-time Tiny with 3s chunks ├── realTinyFeed.csv # Metrics for real-time Tiny with feedback └── realTinyVad.csv # Metrics for real-time Tiny with VAD File Count by Configuration Configuration Number of Files batchBase 159 batchLarge 159 batchTiny 159 realBase2s 159 realBase3s 159 realBaseFeed 159 realBaseVad 159 realTiny2s 159 realTiny3s 159 realTinyFeed 159 realTinyVad 158 Total 1,748 Audio Files The transcribed audio files are from the Gigaspeech corpus and are identified by their unique IDs: POD prefix: Podcast audio files (e.g., POD1000000004.txt) YOU prefix: YouTube audio files (e.g., YOU1000000000.txt) The original audio files are available in the Gigaspeech repository and are not included in this dataset. Transcription Formats Batch Transcriptions Batch mode transcriptions (offline processing) are stored as plain text files containing the complete transcription: Hey friends, I don't know about you, but when I was a kid I knew absolutely nothing about money. Well there's a new show from Marketplace and Brains on that will help address just that... Real-time Transcriptions Real-time mode transcriptions (online processing) are stored in JSON format with sentence-level timestamps: {"sentence": "Friends, I don't know about you, but when I was a kid.", "timestamp": 3325}; {"sentence": "I knew absolutely nothing about it.", "timestamp": 5223}; {"sentence": "money. Well, there's a new show.", "timestamp": 7280}; sentence: The transcribed text segment timestamp: Time in milliseconds from the start of audio playback when the transcription was presented to the user Evaluation Metrics Each CSV file in the results/ directory contains evaluation metrics for the corresponding experimental configuration. The metrics include: Metric Description FILE Audio file identifier from Gigaspeech WER Word Error Rate - measures word-level accuracy MER Match Error Rate - alternative error metric WIL Word Information Lost - information-theoretic measure Example Results FILE,WER,MER,WIL POD1000000039,0.2107904642409034,0.1969519343493552,0.2719032632574976 YOU1000000107,0.19279907084785133,0.18485523385300667,0.23827720179632295 POD1000000011,0.11490082244799225,0.11121517209084524,0.1535611413001976 Lower values indicate better transcription accuracy. Experimental Configurations Transcription Modes Batch (Offline): The entire audio file is processed at once before transcription begins. This represents the ideal scenario where the complete audio is available. Real-time (Online): Audio is processed in a streaming fashion, simulating real-time transcription scenarios where audio chunks are transcribed as they arrive. Whisper Models Tiny: Fastest, smallest model (~39M parameters) Base: Balanced speed and accuracy (~74M parameters) Large: Highest accuracy, slowest (~1550M parameters) Audio Splitting Algorithms (Real-time only) VAD (Voice Activity Detection): Segments audio based on detected speech activity Fixed-length (2s): Splits audio into fixed 2-second chunks Fixed-length (3s): Splits audio into fixed 3-second chunks Feedback: Adaptive splitting based on transcription feedback Ground Truth Ground truth transcriptions are obtained from the Gigaspeech repository metadata. The Gigaspeech corpus provides professionally annotated transcriptions for all audio files. Usage Examples Loading Batch Transcriptions # Read a batch transcription with open('transcriptions/batchBase/POD1000000004.txt', 'r') as f: transcription = f.read() print(transcription) Loading Real-time Transcriptions import json # Read real-time transcription with timestamps with open('transcriptions/realBase2s/POD1000000004.txt', 'r') as f: lines = f.readlines() # Parse JSON entries transcriptions = [json.loads(line.rstrip(';')) for line in lines] for entry in transcriptions: print(f"[{entry['timestamp']}ms] {entry['sentence']}") Loading Evaluation Metrics import pandas as pd # Load metrics for batch Base model metrics = pd.read_csv('results/batchBase.csv') # Display summary statistics print(metrics.describe()) # Find best and worst performing files best_file = metrics.loc[metrics['WER'].idxmin()] worst_file = metrics.loc[metrics['WER'].idxmax()] print(f"Best WER: {best_file['FILE']} ({best_file['WER']:.4f})") print(f"Worst WER: {worst_file['FILE']} ({worst_file['WER']:.4f})") Comparing Models import pandas as pd import matplotlib.pyplot as plt # Load metrics for different models batch_tiny = pd.read_csv('results/batchTiny.csv') batch_base = pd.read_csv('results/batchBase.csv') batch_large = pd.read_csv('results/batchLarge.csv') # Compare average WER models = ['Tiny', 'Base', 'Large'] wer_means = [ batch_tiny['WER'].mean(), batch_base['WER'].mean(), batch_large['WER'].mean() ] plt.bar(models, wer_means) plt.ylabel('Average WER') plt.title('Whisper Model Comparison') plt.show() File Naming Convention POD[ID].txt: Transcriptions from podcast audio files YOU[ID].txt: Transcriptions from YouTube audio files Where [ID] is a unique 10-digit identifier from the Gigaspeech corpus FAIR Compliance This dataset follows the FAIR (Findable, Accessible, Interoperable, Reusable) principles for scientific data management: Findable Persistent Identifier: The dataset is deposited in Zenodo with a Digital Object Identifier (DOI) for permanent citation and retrieval Rich Metadata: Comprehensive metadata including title, authors, keywords, and abstract describing the dataset Searchable: Indexed in multiple research data repositories and discoverable through standard search engines Unique Identifiers: Each audio file maintains its unique Gigaspeech ID for cross-referencing with the original corpus Accessible Open Access: Freely available for download without registration barriers Standard Protocols: Accessible via HTTPS and standard web protocols Long-term Preservation: Hosted on Zenodo, ensuring long-term availability and preservation Complete Documentation: This README provides comprehensive documentation of file formats, structure, and usage Interoperable Standard Formats: Transcriptions stored in plain text (.txt) and JSON formats Metrics stored in CSV format for easy import into analysis tools All formats are non-proprietary and widely supported Consistent Schema: Uniform file structure across all experimental configurations Cross-referencing: File IDs link directly to the Gigaspeech corpus for audio retrieval Standardized Metrics: WER, MER, and WIL are widely recognized ASR evaluation metrics Reusable Clear Licensing: Dataset released under an open license (specify in repository) Detailed Provenance: Source data clearly identified (Gigaspeech corpus) Processing methods documented (Whisper models, versions, parameters) Experimental conditions fully described Quality Assurance: Ground truth from professionally annotated Gigaspeech transcriptions Comprehensive Documentation: File formats and structures clearly explained Usage examples provided in multiple programming languages Citation information for proper attribution Research Paper: Accompanied by peer-reviewed publication detailing methodology and findings Metadata Summary Attribute Value Title Assessing Latency in ASR Systems: Whisper Transcription Dataset Creators Carlos Arriaga, Alejandro Pozo, Javier Conde, Alvaro Alonso Publication Date 2025 Publisher Zenodo Resource Type Dataset Subject Area Speech Recognition, Natural Language Processing, Machine Learning Language English (transcriptions) Source Data Gigaspeech Corpus File Formats .txt, .json, .csv Size 1,748 files Related Publication IEEE Internet Computing, DOI: 10.1109/MIC.2025.3614363 Data Statistics Total transcription files: 1,748 Unique audio files: 159 Experimental configurations: 11 Whisper models tested: 3 (Tiny, Base, Large) Splitting algorithms: 4 (VAD, 2s, 3s, Feedback) Transcription modes: 2 (Batch, Real-time)