add and bless AV1 video encoding/decoding package #998

New issue

Open

opened 2023-09-12 18:06:06 +00:00 by emidoots · 2 comments

emidoots commented

2023-09-12 18:06:06 +00:00

(Migrated from github.com)

Similar to how we have mach-flac and mach-opus, which are going to be our blessed audio formats, we should have bindings for encoding and decoding AV1 video as our blessed video format.

❤️ 2

mateli commented

2023-10-18 09:04:31 +00:00

(Migrated from github.com)

When generating video from tools like blender we have the option of creating some compact vector based format that do not need further compression. Simply by not having pre-rendered video and instead rendering it while playing.

It could also be possible to use AI to deconstruct rendered video into a compact vector based format.

Do we support hardware encode/decode? Zero-copy when done on the GPU (dma-buf)? Other interesting compression news. Perhaps AI can be used to achieve a much higher compression ratio than AV1 and other traditional codecs. https://hexus.net/tech/news/software/146086-nvidia-uses-ai-reduce-video-conferencing-data-use-10x/ When generating video from tools like blender we have the option of creating some compact vector based format that do not need further compression. Simply by not having pre-rendered video and instead rendering it while playing. It could also be possible to use AI to deconstruct rendered video into a compact vector based format.

emidoots commented

2023-11-06 04:01:03 +00:00

(Migrated from github.com)

Probably we'll go with https://gitlab.com/AOMediaCodec/SVT-AV1 - it looks quite good, C99, has an encoder and decoder, seems fast (built by Intel and Netflix), etc.

Other options either don't offer an encoder (we need one), or would require a C++ and/or Rust compiler.

Do we support hardware encode/decode? Zero-copy when done on the GPU (dma-buf)?

Most libraries for this I think are software-only currently, AV1 hardware support is not widespread yet. There exist extensions for Vulkan, Metal, and d3d12 that we can leverage long-term, but this will need to be done in sysgpu as an extension/optional feature until hardware support is broad enough.

Probably we'll go with https://gitlab.com/AOMediaCodec/SVT-AV1 - it looks quite good, C99, has an encoder and decoder, seems fast (built by Intel and Netflix), etc. Other options either don't offer an encoder (we need one), or would require a C++ and/or Rust compiler. > Do we support hardware encode/decode? Zero-copy when done on the GPU (dma-buf)? Most libraries for this I think are software-only currently, AV1 hardware support is not widespread yet. There exist extensions for Vulkan, Metal, and d3d12 that we can leverage long-term, but this will need to be done in [sysgpu](https://machengine.org/pkg/mach-sysgpu/) as an extension/optional feature until hardware support is broad enough.