mach: add blessed texture format(s) #1107

New issue

Open

opened 2023-11-06 04:09:39 +00:00 by emidoots · 3 comments

emidoots commented

2023-11-06 04:09:39 +00:00

(Migrated from github.com)

Lossless format: png
Lossy format: TBD
- So far the plan was to use basisu 'supercompressed textures' in either .basis or .KTX2 format: https://github.com/BinomialLLC/basis_universal
- Support is probably broad enough for BC7 on desktop platforms, and ASTC for everything else these days. But we would need a fallback software decoder. And also an encoder for each.
- .ktx2 format is quite nasty, if you look at the file format it is pretty convoluted. .basis is fairly nice and simple.
- basis universal has been updated with UASTC support:

The system now supports two modes: a high quality mode which is internally based off the UASTC compressed texture format, and the original lower quality mode which is based off a subset of ETC1 called "ETC1S". UASTC is for extremely high quality (similar to BC7 quality) textures, and ETC1S is for very small files. The ETC1S system includes built-in data compression, while the UASTC system includes an optional Rate Distortion Optimization (RDO) post-process stage that conditions the encoded UASTC texture data in the .basis file so it can be more effectively LZ compressed by the end user. More technical details about UASTC integration are here.

Perhaps we can use that with just .basis files?

* Lossless format: png * Lossy format: TBD * So far the plan was to use [basisu](https://github.com/hexops/mach/issues/965) 'supercompressed textures' in either `.basis` or `.KTX2` format: https://github.com/BinomialLLC/basis_universal * Support is probably broad enough for BC7 on desktop platforms, and ASTC for everything else these days. But we would need a fallback software decoder. And also an encoder for each. * `.ktx2` format is quite nasty, if you look at the file format it is pretty convoluted. `.basis` is fairly nice and simple. * basis universal has been updated with UASTC support: > The system now supports two modes: a high quality mode which is internally based off the [UASTC compressed texture format](https://richg42.blogspot.com/2020/01/uastc-block-format-encoding.html), and the original lower quality mode which is based off a subset of ETC1 called "ETC1S". UASTC is for extremely high quality (similar to BC7 quality) textures, and ETC1S is for very small files. The ETC1S system includes built-in data compression, while the UASTC system includes an optional Rate Distortion Optimization (RDO) post-process stage that conditions the encoded UASTC texture data in the .basis file so it can be more effectively LZ compressed by the end user. More technical details about UASTC integration are [here](https://github.com/BinomialLLC/basis_universal/wiki/UASTC-implementation-details). Perhaps we can use that with just `.basis` files?

pdoane commented

2023-11-06 05:00:37 +00:00

(Migrated from github.com)

In the future it could be useful to have platform optimized formats. Having a universal solution is important though and the thinking here sounds good.

anyputer commented

2024-07-15 13:12:01 +00:00

(Migrated from github.com)

KTX2, WebP, JPEG XL are solid options IMO.

PNG

Pros:

Memory safe decoding with the wuffs library, usable with Zig.
Best existing tooling.

Cons:

Subpar encoding speed, compared to libjxl's fast_lossless implementation.
Subpar compression compared to WebP lossless.

WebP

Pros:

Best lossless compression I've tested (cwebp -z 9 -o. Great at pixel art.
Its lossless codec looks reasonable to implement a decoder for.
Better tooling support than other formats that are mentioned that aren't PNG.

Cons:

Higher than 8 bits of color depth is not supported.
Width and height are limited to 16384 (14 bit + 1). This is actually the most common GPU texture size limit.
Its lossy mode isn't that efficient, and is based on VP8 instead of a format modern GPUs support (AV1?).
AVIF is way better at lossy, although the best GPU compressed format it can do is YUV420.
Notable CVE

JPEG XL

Pros:

Lossless (cjxl -e 1 -d 0) is extremely fast to encode and decode on my machine, optimized for multiple cores.
It loves to beat QOI in speed and resulting compression.
Bit depths up to 32 bit are supported.
Supports many channels. This can be used for storing many PBR maps in a single file.
Decent tooling support in 2024. Not (yet) supported in Blender though.

Cons:

Format is pretty complex.
Consider looking at the Rust zune library for assessing complexity?
Its lossy support doesn't seem useful due to missing GPU texture compression.

QOI

Pros:

Format is designed to be simple to write an encoder and decoder for, with minimal code binary size.
Plenty fast to encode and decode. However cjxl -d 0 -e 1 is still king.

Cons:

Higher than 8 bits of color depth is not supported.
Limited tooling support, but is gaining popularity.

KTX2

Pros:

A single container format for both lossless and BasisU, two for the price of one.
This container format is relatively easy to parse, with little endian 32-bit aligned integers.
May benefit wasm32 for code size :)
Supports lossless compression with ZSTD, being fast to decode on a single thread.
Any GPU-supported pixel format is allowed by the format, with Vulkan behind it.
Zig standard library supports ZSTD decoding, however doesn't work with every ZSTD data I've tried yet...
Multi-threaded encoding of ZSTD is supported by libzstd, however not in the Zig standard library.
The BasisU variant is compatible with glTF.
Artists can work with existing glTF exporters.

Cons:

Limited tooling support, less than QOI.
Can be improved in the future if Mach were to push for it.
Multi-threaded decode is not feasible.
However, a format that can quickly be decoded on a single thread could in theory be helpful for a game loading many images at once, one image per thread.
Supports more formats than WebGPU knows to do with, including RGB565.

KTX2, WebP, JPEG XL are solid options IMO. ## PNG Pros: - Memory safe decoding with the `wuffs` library, usable with Zig. - Best existing tooling. Cons: - Subpar encoding speed, compared to `libjxl`'s fast_lossless implementation. - Subpar compression compared to WebP lossless. ## WebP Pros: - Best lossless compression I've tested (`cwebp -z 9 -o`. Great at pixel art. - Its lossless codec looks reasonable to implement a decoder for. - Better tooling support than other formats that are mentioned that aren't PNG. Cons: - Higher than 8 bits of color depth is not supported. - Width and height are limited to 16384 (14 bit + 1). [This is actually the most common GPU texture size limit.](https://vulkan.gpuinfo.org/displaydevicelimit.php?name=maxImageDimension2D&platform=all) - Its lossy mode isn't that efficient, and is based on VP8 instead of a format modern GPUs support (AV1?). AVIF is way better at lossy, although the best GPU compressed format it can do is YUV420. - [Notable CVE](https://blog.cloudflare.com/uncovering-the-hidden-webp-vulnerability-cve-2023-4863) ## [JPEG XL](https://www.youtube.com/watch?v=VJaa1Le4W7c) Pros: - Lossless (`cjxl -e 1 -d 0`) is extremely fast to encode and decode on my machine, optimized for multiple cores. It loves to beat QOI in speed and resulting compression. - Bit depths up to 32 bit are supported. - Supports many channels. This can be used for storing many PBR maps in a single file. - Decent tooling support in 2024. Not (yet) supported in Blender though. Cons: - Format is pretty complex. Consider looking at the Rust `zune` library for assessing complexity? - Its lossy support doesn't seem useful due to missing GPU texture compression. ## QOI Pros: - Format is designed to be simple to write an encoder and decoder for, with minimal code binary size. - Plenty fast to encode and decode. However `cjxl -d 0 -e 1` is still king. Cons: - Higher than 8 bits of color depth is not supported. - Limited tooling support, but is gaining popularity. ## KTX2 Pros: - A single container format for both lossless and BasisU, two for the price of one. - This container format is relatively easy to parse, with little endian 32-bit aligned integers. May benefit wasm32 for code size :) - Supports lossless compression with ZSTD, being fast to decode on a single thread. - Any GPU-supported pixel format is allowed by the format, with Vulkan behind it. - Zig standard library supports ZSTD decoding, however doesn't work with every ZSTD data I've tried yet... - Multi-threaded encoding of ZSTD is supported by `libzstd`, however not in the Zig standard library. - The BasisU variant is compatible with glTF. Artists can work with existing glTF exporters. Cons: - Limited tooling support, less than QOI. Can be improved in the future if Mach were to push for it. - Multi-threaded decode is not feasible. However, a format that can quickly be decoded on a single thread could in theory be helpful for a game loading many images at once, one image per thread. - Supports more formats than WebGPU knows to do with, including RGB565.

emidoots commented

2024-07-15 17:54:47 +00:00

(Migrated from github.com)

PNG, WebP, JPEG, and others are non-options because they are only disk-compressed.

KTX2 is likely a non-option because the spec for it is nasty.

PNG, WebP, JPEG, and others are non-options because they are only disk-compressed. KTX2 is likely a non-option because the spec for it is nasty.