Applying machine learning to identify optimal file compression methods

This independent study addresses the inefficiency of using a single lossless compression algorithm for diverse file types, a common practice in financial reporting and other domains. We propose an adaptive framework that uses supervised machine learning to predict the most suitable compression method for each file. A dataset of approximately 120,000 real-world files (including text, tabular, and semi-structured formats) was created. Each file was compressed using six major algorithms (Zstd, LZ4, Brotli, LZMA, Bzip2, and zlib) to determine the "ground-truth" best method based on the lowest compression ratio achieved within a 30-second time limit. We extracted an initial set of 15 structural features for each file. A Sequential Feature Selection (SFS) technique was then employed to identify the most predictive subset of features. The final model predicts the optimal algorithm, achieving compression ratios close to the empirical optimum without the high cost of an exhaustive search. This model can be embedded into existing data pipelines to automatically reduce storage costs and data transfer times with minimal added latency.