OpenZL

OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. It takes a description of your data and builds a specialized compressor optimized for your specific format.

The framework consists of a core library and tools to generate these specialized compressors, which are all compatible with a single universal decompressor. It is designed for engineers that deal with large quantities of specialized datasets, such as AI workloads, and require high speed for their processing pipelines.

Here is a performance comparison:

| Compressor | Ratio | Compression Speed | Decompression Speed | | :--- | :--- | :--- | :--- | | zstd -3 | x1.31 | 115 MB/s | 890 MB/s | | xz -9 | x1.64 | 3.1 MB/s | 30 MB/s | | OpenZL | x2.06 | 203 MB/s | 822 MB/s |

The result is much stronger ratios than generic compressors, at the speeds required by datacenter workloads.