Developer Utilities Reference Data: Token Costs, Model Pricing, and Text Processing Benchmarks

Reference datasets compiled by Mohit Khare for developer utilities and tooling. This dataset provides three categories of structured data useful for software engineers working with large language models and text processing systems. Contents include: (1) Comparative pricing data for major LLM APIs including Anthropic Claude, OpenAI GPT, Google Gemini, Meta Llama, Mistral, and DeepSeek, with per-million-token costs and context window sizes. (2) Empirical token estimation benchmarks measuring character-to-token and word-to-token ratios across tokenizers, plus language-specific multipliers for 10 languages. (3) Performance benchmarks for common text processing operations (tokenization, NER, regex, JSON serialization) across popular Python libraries. Related resources: mohitkhare.me - Developer portfolio and utilities Blog - Technical writing on development topics